Commit graph

564 commits

Author SHA1 Message Date
puni9869 cafce3b4b5
Allow to archive labels (#26478)
## Archived labels 

This adds the structure to allow for archived labels.
Archived labels are, just like closed milestones or projects, a medium to hide information without deleting it.
It is especially useful if there are outdated labels that should no longer be used without deleting the label entirely.

## Changes

1. UI and API have been equipped with the support to mark a label as archived
2. The time when a label has been archived will be stored in the DB

## Outsourced for the future

There's no special handling for archived labels at the moment.
This will be done in the future.

## Screenshots

![image](https://github.com/go-gitea/gitea/assets/80308335/208f95cd-42e4-4ed7-9a1f-cd2050a645d4)

![image](https://github.com/go-gitea/gitea/assets/80308335/746428e0-40bb-45b3-b992-85602feb371d)

Part of https://github.com/go-gitea/gitea/issues/25237

---------

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-08-14 11:56:14 +02:00
delvh e10ba5e5e0
Rename Sync2 -> Sync (#26479)
The xorm `Sync2` has already been deprecated in favor of `Sync`,
so let's do the same inside the Gitea codebase.

Command used to replace everything:
```sh
for i in $(ag Sync2 --files-with-matches); do vim $i -c ':%sno/Sync2/Sync/g' -c ':wq'; done
```
2023-08-13 21:17:21 +02:00
wxiaoguang 82ea557dd3
Fix stderr usages (#26477) 2023-08-13 20:49:30 +08:00
KN4CK3R d08ab7be9c
Fix typo in metadata (#26207)
https://github.com/go-gitea/gitea/pull/26194#discussion_r1277094525

There is no need to backport because these names are just used internal.
2023-07-30 23:54:22 +00:00
KN4CK3R 7866a6e0e2
Prevent primary key update on migration (#26192)
Fixes #25918

The migration fails on MSSQL because xorm tries to update the primary
key column. xorm prevents this if the column is marked as auto
increment:

c622cdaf89/internal/statements/update.go (L38-L40)

I think it would be better if xorm would check for primary key columns
here because updating such columns is bad practice. It looks like if
that auto increment check should do the same.

fyi @lunny
2023-07-28 09:54:31 +02:00
yp05327 74ca4377ff
Drop the correct deleted branch table (#26028)
There's a typo in #22743
The correct table name is `deleted_branch` not `deleted_branches`
2023-07-26 08:41:33 +08:00
Bo-Yi Wu 44781f9f5c
Implement auto-cancellation of concurrent jobs if the event is push (#25716)
- cancel running jobs if the event is push
- Add a new function `CancelRunningJobs` to cancel all running jobs of a
run
- Update `FindRunOptions` struct to include `Ref` field and update its
condition in `toConds` function
- Implement auto cancellation of running jobs in the same workflow in
`notify` function

related task: https://github.com/go-gitea/gitea/pull/22751/

---------

Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Signed-off-by: appleboy <appleboy.tw@gmail.com>
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
2023-07-25 11:15:55 +08:00
sillyguodong f5c7d4cfdd
Reduce unnecessary DB queries for Actions tasks (#25199)
Close #24544

Changes:

- Create `action_tasks_version` table to store the latest version of
each scope (global, org and repo).
- When a job with the status of `waiting` is created, the tasks version
of the scopes it belongs to will increase.
- When the status of a job already in the database is updated to
`waiting`, the tasks version of the scopes it belongs to will increase.
- On Gitea side, in `FeatchTask()`, will try to query the
`action_tasks_version` record of the scope of the runner that call
`FetchTask()`. If the record does not exist, will insert a row. Then,
Gitea will compare the version passed from runner to Gitea with the
version in database, if inconsistent, try pick task. Gitea always
returns the latest version from database to the runner.

Related:

- Protocol: https://gitea.com/gitea/actions-proto-def/pulls/10
- Runner: https://gitea.com/gitea/act_runner/pulls/219
2023-07-24 06:11:27 +00:00
caicandong 840830b655
Remove commit status running and warning to align GitHub (#25839)
Fix #25776. Close #25826.

In the discussion of #25776, @wolfogre's suggestion was to remove the
commit status of `running` and `warning` to keep it consistent with
github.

references:
-
https://docs.github.com/en/rest/commits/statuses?apiVersion=2022-11-28#about-commit-statuses


## ⚠️ BREAKING ⚠️

So the commit status of Gitea will be consistent with GitHub, only
`pending`, `success`, `error` and `failure`, while `warning` and
`running` are not supported anymore.

---------

Co-authored-by: Jason Song <i@wolfogre.com>
2023-07-21 16:24:36 +08:00
FuXiaoHei f3d293d2bb
Actions Artifacts support uploading multiple files and directories (#24874)
current actions artifacts implementation only support single file
artifact. To support multiple files uploading, it needs:

- save each file to each db record with same run-id, same artifact-name
and proper artifact-path
- need change artifact uploading url without artifact-id, multiple files
creates multiple artifact-ids
- support `path` in download-artifact action. artifact should download
to `{path}/{artifact-path}`.
- in repo action view, it provides zip download link in artifacts list
in summary page, no matter this artifact contains single or multiple
files.
2023-07-21 10:42:01 +08:00
silverwind 887a683af9
Update tool dependencies, lock govulncheck and actionlint (#25655)
- Update all tool dependencies
- Lock `govulncheck` and `actionlint` to their latest tags

---------

Co-authored-by: 6543 <m.huber@kithara.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-07-09 11:58:06 +00:00
6543 8995046110
Less naked returns (#25713)
just a step towards  #25655

and some related refactoring
2023-07-07 05:31:56 +00:00
silverwind 88f835192d
Replace interface{} with any (#25686)
Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`.

Basically the same [as golang did](2580d0e08d).
2023-07-04 18:36:08 +00:00
Lunny Xiao 6e19484f4d
Sync branches into databases (#22743)
Related #14180
Related #25233 
Related #22639
Close #19786
Related #12763 

This PR will change all the branches retrieve method from reading git
data to read database to reduce git read operations.

- [x] Sync git branches information into database when push git data
- [x] Create a new table `Branch`, merge some columns of `DeletedBranch`
into `Branch` table and drop the table `DeletedBranch`.
- [x] Read `Branch` table when visit `code` -> `branch` page
- [x] Read `Branch` table when list branch names in `code` page dropdown
- [x] Read `Branch` table when list git ref compare page
- [x] Provide a button in admin page to manually sync all branches.
- [x] Sync branches if repository is not empty but database branches are
empty when visiting pages with branches list
- [x] Use `commit_time desc` as the default FindBranch order by to keep
consistent as before and deleted branches will be always at the end.

---------

Co-authored-by: Jason Song <i@wolfogre.com>
2023-06-29 10:03:20 +00:00
a1012112796 4aba8a6a5f
Split lfs size from repository size (#22900)
releated to #21820

- Split `Size` in repository table as two new colunms, one is `GitSize`
for git size, the other is `LFSSize` for lfs data. still store full size
in `Size` colunm.
- Show full size on ui, but show each of them by a `title`;  example:

![image](https://user-images.githubusercontent.com/25342410/218636251-e200f085-d7e7-4a25-9ff1-b586a63e07a9.png)

- Return full size in api response.

---------

Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: DmitryFrolovTri <23313323+DmitryFrolovTri@users.noreply.github.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-06-28 22:41:02 +00:00
Zettat123 48e5a74f21
Support pull_request_target event (#25229)
Fix #25088

This PR adds the support for
[`pull_request_target`](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target)
workflow trigger. `pull_request_target` is similar to `pull_request`,
but the workflow triggered by the `pull_request_target` event runs in
the context of the base branch of the pull request rather than the head
branch. Since the workflow from the base is considered trusted, it can
access the secrets and doesn't need approvals to run.
2023-06-26 14:33:18 +08:00
wxiaoguang 2cdf260f42
Refactor path & config system (#25330)
# The problem

There were many "path tricks":

* By default, Gitea uses its program directory as its work path
* Gitea tries to use the "work path" to guess its "custom path" and
"custom conf (app.ini)"
* Users might want to use other directories as work path
* The non-default work path should be passed to Gitea by GITEA_WORK_DIR
or "--work-path"
* But some Gitea processes are started without these values
    * The "serv" process started by OpenSSH server
    * The CLI sub-commands started by site admin
* The paths are guessed by SetCustomPathAndConf again and again
* The default values of "work path / custom path / custom conf" can be
changed when compiling

# The solution

* Use `InitWorkPathAndCommonConfig` to handle these path tricks, and use
test code to cover its behaviors.
* When Gitea's web server runs, write the WORK_PATH to "app.ini", this
value must be the most correct one, because if this value is not right,
users would find that the web UI doesn't work and then they should be
able to fix it.
* Then all other sub-commands can use the WORK_PATH in app.ini to
initialize their paths.
* By the way, when Gitea starts for git protocol, it shouldn't output
any log, otherwise the git protocol gets broken and client blocks
forever.

The "work path" priority is: WORK_PATH in app.ini > cmd arg --work-path
> env var GITEA_WORK_DIR > builtin default

The "app.ini" searching order is: cmd arg --config > cmd arg "work path
/ custom path" > env var "work path / custom path" > builtin default


## ⚠️ BREAKING

If your instance's "work path / custom path / custom conf" doesn't meet
the requirements (eg: work path must be absolute), Gitea will report a
fatal error and exit. You need to set these values according to the
error log.



----

Close #24818
Close #24222
Close #21606
Close #21498
Close #25107
Close #24981
Maybe close #24503

Replace #23301
Replace #22754

And maybe more
2023-06-21 13:50:26 +08:00
sillyguodong 35a653d7ed
Support configuration variables on Gitea Actions (#24724)
Co-Author: @silverwind @wxiaoguang 
Replace: #24404

See:
- [defining configuration variables for multiple
workflows](https://docs.github.com/en/actions/learn-github-actions/variables#defining-configuration-variables-for-multiple-workflows)
- [vars
context](https://docs.github.com/en/actions/learn-github-actions/contexts#vars-context)

Related to:
- [x] protocol: https://gitea.com/gitea/actions-proto-def/pulls/7
- [x] act_runner: https://gitea.com/gitea/act_runner/pulls/157
- [x] act: https://gitea.com/gitea/act/pulls/43

#### Screenshoot
Create Variable:

![image](https://user-images.githubusercontent.com/33891828/236758288-032b7f64-44e7-48ea-b07d-de8b8b0e3729.png)


![image](https://user-images.githubusercontent.com/33891828/236758174-5203f64c-1d0e-4737-a5b0-62061dee86f8.png)

Workflow:
```yaml
  test_vars:
    runs-on: ubuntu-latest
    steps:
      - name: Print Custom Variables
        run: echo "${{ vars.test_key }}"
      - name: Try to print a non-exist var
        run: echo "${{ vars.NON_EXIST_VAR }}"
```

Actions Log:

![image](https://user-images.githubusercontent.com/33891828/236759075-af0c5950-368d-4758-a8ac-47a96e43b6e2.png)

---
This PR just implement the org / user (depends on the owner of the
current repository) and repo level variables, The Environment level
variables have not been implemented.
Because
[Environment](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#about-environments)
is a module separate from `Actions`. Maybe it would be better to create
a new PR to do it.

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-06-20 22:54:15 +00:00
yp05327 3afc3e4a71
Add missing v in migrations.go (#25252) 2023-06-14 12:35:35 +00:00
sillyguodong 020ab3d4c8
Fix description of drop custom_labels migration (#25243)
Follow: #24806
2023-06-14 04:46:55 +00:00
Lunny Xiao d6dd6d641b
Fix all possible setting error related storages and added some tests (#23911)
Follow up #22405

Fix #20703 

This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.

- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-06-14 11:42:38 +08:00
sillyguodong 8228751c55
Support changing labels of Actions runner without re-registration (#24806)
close #24540

related:
- Protocol: https://gitea.com/gitea/actions-proto-def/pulls/9
- Runner side: https://gitea.com/gitea/act_runner/pulls/201

changes:
- Add column of `labels` to table `action_runner`, and combine the value
of `agent_labels` and `custom_labels` column to `labels` column.
- Store `labels` when registering `act_runner`.
- Update `labels` when `act_runner` starting and calling `Declare`.
- Users cannot modify the `custom labels` in edit page any more.

other changes:
- Store `version` when registering `act_runner`.
- If runner is latest version, parse version from `Declare`. But older
version runner still parse version from request header.
2023-06-13 22:28:31 +08:00
Jack Hay 18de83b2a3
Redesign Scoped Access Tokens (#24767)
## Changes
- Adds the following high level access scopes, each with `read` and
`write` levels:
    - `activitypub`
    - `admin` (hidden if user is not a site admin)
    - `misc`
    - `notification`
    - `organization`
    - `package`
    - `issue`
    - `repository`
    - `user`
- Adds new middleware function `tokenRequiresScopes()` in addition to
`reqToken()`
  -  `tokenRequiresScopes()` is used for each high-level api section
- _if_ a scoped token is present, checks that the required scope is
included based on the section and HTTP method
  - `reqToken()` is used for individual routes
- checks that required authentication is present (but does not check
scope levels as this will already have been handled by
`tokenRequiresScopes()`
- Adds migration to convert old scoped access tokens to the new set of
scopes
- Updates the user interface for scope selection

### User interface example
<img width="903" alt="Screen Shot 2023-05-31 at 1 56 55 PM"
src="https://github.com/go-gitea/gitea/assets/23248839/654766ec-2143-4f59-9037-3b51600e32f3">
<img width="917" alt="Screen Shot 2023-05-31 at 1 56 43 PM"
src="https://github.com/go-gitea/gitea/assets/23248839/1ad64081-012c-4a73-b393-66b30352654c">

## tokenRequiresScopes  Design Decision
- `tokenRequiresScopes()` was added to more reliably cover api routes.
For an incoming request, this function uses the given scope category
(say `AccessTokenScopeCategoryOrganization`) and the HTTP method (say
`DELETE`) and verifies that any scoped tokens in use include
`delete:organization`.
- `reqToken()` is used to enforce auth for individual routes that
require it. If a scoped token is not present for a request,
`tokenRequiresScopes()` will not return an error

## TODO
- [x] Alphabetize scope categories
- [x] Change 'public repos only' to a radio button (private vs public).
Also expand this to organizations
- [X] Disable token creation if no scopes selected. Alternatively, show
warning
- [x] `reqToken()` is missing from many `POST/DELETE` routes in the api.
`tokenRequiresScopes()` only checks that a given token has the correct
scope, `reqToken()` must be used to check that a token (or some other
auth) is present.
   -  _This should be addressed in this PR_
- [x] The migration should be reviewed very carefully in order to
minimize access changes to existing user tokens.
   - _This should be addressed in this PR_
- [x] Link to api to swagger documentation, clarify what
read/write/delete levels correspond to
- [x] Review cases where more than one scope is needed as this directly
deviates from the api definition.
   - _This should be addressed in this PR_
   - For example: 
   ```go
	m.Group("/users/{username}/orgs", func() {
		m.Get("", reqToken(), org.ListUserOrgs)
		m.Get("/{org}/permissions", reqToken(), org.GetUserOrgsPermissions)
}, tokenRequiresScopes(auth_model.AccessTokenScopeCategoryUser,
auth_model.AccessTokenScopeCategoryOrganization),
context_service.UserAssignmentAPI())
   ```

## Future improvements
- [ ] Add required scopes to swagger documentation
- [ ] Redesign `reqToken()` to be opt-out rather than opt-in
- [ ] Subdivide scopes like `repository`
- [ ] Once a token is created, if it has no scopes, we should display
text instead of an empty bullet point
- [ ] If the 'public repos only' option is selected, should read
categories be selected by default

Closes #24501
Closes #24799

Co-authored-by: Jonathan Tran <jon@allspice.io>
Co-authored-by: Kyle D <kdumontnu@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
2023-06-04 20:57:16 +02:00
JakobDev aaa1094663
Add the ability to pin Issues (#24406)
This adds the ability to pin important Issues and Pull Requests. You can
also move pinned Issues around to change their Position. Resolves #2175.

## Screenshots

![grafik](https://user-images.githubusercontent.com/15185051/235123207-0aa39869-bb48-45c3-abe2-ba1e836046ec.png)

![grafik](https://user-images.githubusercontent.com/15185051/235123297-152a16ea-a857-451d-9a42-61f2cd54dd75.png)

![grafik](https://user-images.githubusercontent.com/15185051/235640782-cbfe25ec-6254-479a-a3de-133e585d7a2d.png)

The Design was mostly copied from the Projects Board.

## Implementation
This uses a new `pin_order` Column in the `issue` table. If the value is
set to 0, the Issue is not pinned. If it's set to a bigger value, the
value is the Position. 1 means it's the first pinned Issue, 2 means it's
the second one etc. This is dived into Issues and Pull requests for each
Repo.

## TODO
- [x] You can currently pin as many Issues as you want. Maybe we should
add a Limit, which is configurable. GitHub uses 3, but I prefer 6, as
this is better for bigger Projects, but I'm open for suggestions.
- [x] Pin and Unpin events need to be added to the Issue history.
- [x] Tests
- [x] Migration

**The feature itself is currently fully working, so tester who may find
weird edge cases are very welcome!**

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-25 15:17:19 +02:00
wxiaoguang 4647660776
Rewrite logger system (#24726)
## ⚠️ Breaking

The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.

Although many legacy options still work, it's encouraged to upgrade to
the new options.

The SMTP logger is deleted because SMTP is not suitable to collect logs.

If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.

## Description

Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)

Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)

There is a new document (with examples): `logging-config.en-us.md`

This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.

## The old problems

The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.


## The new design

See `logger.go` for documents.


## Screenshot

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff)


![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9)


![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee)

</details>

## TODO

* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-21 22:35:11 +00:00
FuXiaoHei c757765a9e
Implement actions artifacts (#22738)
Implement action artifacts server api.

This change is used for supporting
https://github.com/actions/upload-artifact and
https://github.com/actions/download-artifact in gitea actions. It can
run sample workflow from doc
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts.
The api design is inspired by
https://github.com/nektos/act/blob/master/pkg/artifacts/server.go and
includes some changes from gitea internal structs and methods.

Actions artifacts contains two parts:

- Gitea server api and storage (this pr implement basic design without
some complex cases supports)
- Runner communicate with gitea server api (in comming)

Old pr https://github.com/go-gitea/gitea/pull/22345 is outdated after
actions merged. I create new pr from main branch.


![897f7694-3e0f-4f7c-bb4b-9936624ead45](https://user-images.githubusercontent.com/2142787/219382371-eb3cf810-e4e0-456b-a8ff-aecc2b1a1032.jpeg)

Add artifacts list in actions workflow page.
2023-05-19 21:37:57 +08:00
wxiaoguang 6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 19:49:59 +08:00
Lunny Xiao 377a0a20f0
Merge setting.InitXXX into one function with options (#24389)
This PR will merge 3 Init functions on setting packages as 1 and
introduce an options struct.
2023-05-04 11:55:35 +08:00
KN4CK3R bf999e4069
Add Debian package registry (#24426)
Co-authored-by: @awkwardbunny

This PR adds a Debian package registry.
You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing.
Source packages are not supported at the moment and I did not find
documentation of the architecture "all" and how these packages should be
treated.


![grafik](https://user-images.githubusercontent.com/1666336/218126879-eb80a866-775c-4c8e-8529-5797203a64e6.png)

Part of #20751.

Revised copy of #22854.

---------

Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-02 12:31:35 -04:00
Yarden Shoham c0ddec8a2a
Revert "Add Debian package registry" (#24412)
Reverts go-gitea/gitea#22854
2023-04-28 18:06:41 -04:00
KN4CK3R bf77e2163b
Add Debian package registry (#22854)
Co-authored-by: @awkwardbunny

This PR adds a Debian package registry. You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing. Source packages are not supported at the
moment and I did not find documentation of the architecture "all" and
how these packages should be treated.

---------

Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2023-04-28 17:51:36 -04:00
JakobDev c41bc4f127
Display when a repo was archived (#22664)
This adds the date a repo is archived to Gitea and shows it in the UI
and API. A feature, that GitHub has been [introduced
recently](https://github.blog/changelog/2022-11-23-repository-archive-date-now-shown-in-ui/).

I currently don't know how to correctly deal with the Date in the
template, as different languages have different ways of writing a date.


![grafik](https://user-images.githubusercontent.com/15185051/234315187-7db5763e-d96e-4080-b894-9be178bfb6e1.png)

---------

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-04-26 10:46:26 -04:00
silverwind 47748df9b3
Enable forbidigo linter (#24278)
Enable [forbidigo](https://github.com/ashanbrown/forbidigo) linter which
forbids print statements. Will check how to integrate this with the
smallest impact possible, so a few `nolint` comments will likely be
required. Plan is to just go through the issues and either:

- Remove the print if it is nonsensical
- Add a `//nolint` directive if it makes sense

I don't plan on investigating the individual issues any further.

<details>
<summary>Initial Lint Results</summary>

```
modules/log/event.go:348:6: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Println(err)

					^

modules/log/event.go:382:6: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Println(err)

					^

modules/queue/unique_queue_disk_channel_test.go:20:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("TempDir %s\n", tmpDir)

	^

contrib/backport/backport.go:168:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* Backporting %s to %s as %s\n", pr, localReleaseBranch, backportBranch)

	^

contrib/backport/backport.go:216:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("* Navigate to %s to open PR\n", url)

			^

contrib/backport/backport.go:223:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `xdg-open %s`\n", url)

	^

contrib/backport/backport.go:233:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `git push -u %s %s`\n", remote, backportBranch)

	^

contrib/backport/backport.go:243:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* Amending commit to prepend `Backport #%s` to body\n", pr)

	^

contrib/backport/backport.go:272:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("* Attempting git cherry-pick --continue")

		^

contrib/backport/backport.go:281:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* Attempting git cherry-pick %s\n", sha)

	^

contrib/backport/backport.go:297:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* Current branch is %s\n", currentBranch)

	^

contrib/backport/backport.go:299:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("* Current branch is %s - not checking out\n", currentBranch)

		^

contrib/backport/backport.go:304:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("* Branch %s already exists. Checking it out...\n", backportBranch)

		^

contrib/backport/backport.go:308:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `git checkout -b %s %s`\n", backportBranch, releaseBranch)

	^

contrib/backport/backport.go:313:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `git fetch %s main`\n", remote)

	^

contrib/backport/backport.go:316:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(string(out))

		^

contrib/backport/backport.go:319:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Println(string(out))

	^

contrib/backport/backport.go:321:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `git fetch %s %s`\n", remote, releaseBranch)

	^

contrib/backport/backport.go:324:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(string(out))

		^

contrib/backport/backport.go:327:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Println(string(out))

	^

models/unittest/fixtures.go:50:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Unsupported RDBMS for integration tests")

		^

models/unittest/fixtures.go:89:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("LoadFixtures failed after retries: %v\n", err)

		^

models/unittest/fixtures.go:110:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("Failed to generate sequence update: %v\n", err)

			^

models/unittest/fixtures.go:117:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Printf("Failed to update sequence: %s Error: %v\n", value, err)

					^

models/migrations/base/tests.go:118:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Environment variable $GITEA_ROOT not set")

		^

models/migrations/base/tests.go:127:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Could not find gitea binary at %s\n", setting.AppPath)

		^

models/migrations/base/tests.go:134:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Environment variable $GITEA_CONF not set - defaulting to %s\n", giteaConf)

		^

models/migrations/base/tests.go:145:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Unable to create temporary data path %v\n", err)

		^

models/migrations/base/tests.go:154:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Unable to InitFull: %v\n", err)

		^

models/migrations/v1_11/v112.go:34:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

				fmt.Printf("Error: %v", err)

				^

contrib/fixtures/fixture_generation.go:36:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("CreateTestEngine: %+v", err)

		^

contrib/fixtures/fixture_generation.go:40:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("PrepareTestDatabase: %+v\n", err)

		^

contrib/fixtures/fixture_generation.go:46:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

				fmt.Printf("generate '%s': %+v\n", r, err)

				^

contrib/fixtures/fixture_generation.go:53:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

				fmt.Printf("generate '%s': %+v\n", g.name, err)

				^

contrib/fixtures/fixture_generation.go:71:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("%s created.\n", path)

			^

services/gitdiff/gitdiff_test.go:543:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	println(result)

	^

services/gitdiff/gitdiff_test.go:560:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	println(result)

	^

services/gitdiff/gitdiff_test.go:577:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	println(result)

	^

modules/web/routing/logger_manager.go:34:2: use of `print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	print Printer

	^

modules/doctor/paths.go:109:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Warning: can't remove temporary file: '%s'\n", tmpFile.Name())

		^

tests/test_utils.go:33:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf(format+"\n", args...)

	^

tests/test_utils.go:61:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Environment variable $GITEA_CONF not set, use default: %s\n", giteaConf)

		^

cmd/actions.go:54:9: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	_, _ = fmt.Printf("%s\n", respText)

	       ^

cmd/admin_user_change_password.go:74:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("%s's password has been successfully updated!\n", user.Name)

	^

cmd/admin_user_create.go:109:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("generated random password is '%s'\n", password)

		^

cmd/admin_user_create.go:164:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Access token was successfully created... %s\n", t.Token)

		^

cmd/admin_user_create.go:167:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("New user '%s' has been successfully created!\n", username)

	^

cmd/admin_user_generate_access_token.go:74:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("%s\n", t.Token)

		^

cmd/admin_user_generate_access_token.go:76:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Access token was successfully created: %s\n", t.Token)

		^

cmd/admin_user_must_change_password.go:56:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("Updated %d users setting MustChangePassword to %t\n", n, mustChangePassword)

	^

cmd/convert.go:44:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Converted successfully, please confirm your database's character set is now utf8mb4")

		^

cmd/convert.go:50:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Converted successfully, please confirm your database's all columns character is NVARCHAR now")

		^

cmd/convert.go:52:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("This command can only be used with a MySQL or MSSQL database")

		^

cmd/doctor.go:104:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(err)

		^

cmd/doctor.go:105:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Check if you are using the right config file. You can use a --config directive to specify one.")

		^

cmd/doctor.go:243:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(err)

		^

cmd/embedded.go:154:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(a.path)

		^

cmd/embedded.go:198:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Using app.ini at", setting.CustomConf)

		^

cmd/embedded.go:217:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("Extracting to %s:\n", destdir)

	^

cmd/embedded.go:253:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("%s already exists; skipped.\n", dest)

		^

cmd/embedded.go:275:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Println(dest)

	^

cmd/generate.go:63:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("%s", internalToken)

	^

cmd/generate.go:66:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("\n")

		^

cmd/generate.go:78:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("%s", JWTSecretBase64)

	^

cmd/generate.go:81:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("\n")

		^

cmd/generate.go:93:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("%s", secretKey)

	^

cmd/generate.go:96:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("\n")

		^

cmd/keys.go:74:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Println(strings.TrimSpace(authorizedString))

	^

cmd/mailer.go:32:4: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Print("warning: Content is empty")

			^

cmd/mailer.go:35:3: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Print("Proceed with sending email? [Y/n] ")

		^

cmd/mailer.go:40:4: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Println("The mail was not sent")

			^

cmd/mailer.go:49:9: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	_, _ = fmt.Printf("Sent %s email(s) to all users\n", respText)

	       ^

cmd/serv.go:147:3: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		println("Gitea: SSH has been disabled")

		^

cmd/serv.go:153:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("error showing subcommand help: %v\n", err)

			^

cmd/serv.go:175:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			println("Hi there! You've successfully authenticated with the deploy key named " + key.Name + ", but Gitea does not provide shell access.")

			^

cmd/serv.go:177:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			println("Hi there! You've successfully authenticated with the principal " + key.Content + ", but Gitea does not provide shell access.")

			^

cmd/serv.go:179:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			println("Hi there, " + user.Name + "! You've successfully authenticated with the key named " + key.Name + ", but Gitea does not provide shell access.")

			^

cmd/serv.go:181:3: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		println("If this is unexpected, please log in with password and setup Gitea under another user.")

		^

cmd/serv.go:196:5: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

				fmt.Print(`{"type":"gitea","version":1}`)

				^

tests/e2e/e2e_test.go:54:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Error initializing test database: %v\n", err)

		^

tests/e2e/e2e_test.go:63:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("util.RemoveAll: %v\n", err)

		^

tests/e2e/e2e_test.go:67:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Unable to remove repo indexer: %v\n", err)

		^

tests/e2e/e2e_test.go:109:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Printf("%v", stdout.String())

					^

tests/e2e/e2e_test.go:110:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Printf("%v", stderr.String())

					^

tests/e2e/e2e_test.go:113:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Printf("%v", stdout.String())

					^

tests/integration/integration_test.go:124:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Error initializing test database: %v\n", err)

		^

tests/integration/integration_test.go:135:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("util.RemoveAll: %v\n", err)

		^

tests/integration/integration_test.go:139:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Unable to remove repo indexer: %v\n", err)

		^

tests/integration/repo_test.go:357:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("%s", resp.Body)

			^
```

</details>

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-04-24 05:50:58 -04:00
Jason Song ac384c4e1d
Support upload outputs and use needs context on Actions (#24230)
See [Defining outputs for
jobs](https://docs.github.com/en/actions/using-jobs/defining-outputs-for-jobs)
and [Example usage of the needs
context](https://docs.github.com/en/actions/learn-github-actions/contexts#example-usage-of-the-needs-context).

Related to:
- [actions-proto-def
#5](https://gitea.com/gitea/actions-proto-def/pulls/5)
- [act_runner #133](https://gitea.com/gitea/act_runner/pulls/133)

<details>
<summary>Tests & screenshots</summary>

Test workflow file:
```yaml
name: outputs
on: push

jobs:
  job1:
    runs-on: ubuntu-latest
    outputs:
      output1: ${{ steps.step1.outputs.output1 }}
      output2: ${{ steps.step2.outputs.output2 }}
    steps:
      - name: step1
        id: step1
        run: |
          date -Is > output1
          cat output1
          echo "output1=$(cat output1)" >> $GITHUB_OUTPUT
      - name: step2
        id: step2
        run: |
          cat /proc/sys/kernel/random/uuid > output2
          cat output2
          echo "output2=$(cat output2)" >> $GITHUB_OUTPUT
  job2:
    needs: job1
    runs-on: ubuntu-latest
    steps:
      - run: echo ${{ needs.job1.outputs.output1 }}
      - run: echo ${{ needs.job1.outputs.output2 }}
      - run: echo ${{ needs.job1.result }}
```

<img width="397" alt="image"
src="https://user-images.githubusercontent.com/9418365/233313322-903e7ebf-49a7-48e2-8c17-95a4581b3284.png">
<img width="385" alt="image"
src="https://user-images.githubusercontent.com/9418365/233313442-30909135-1711-4b78-a5c6-133fcc79f47c.png">



</details>

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-04-22 16:12:41 -04:00
wxiaoguang 911975059a
Improve test logger (#24235)
Before, there was a `log/buffer.go`, but that design is not general, and
it introduces a lot of irrelevant `Content() (string, error) ` and
`return "", fmt.Errorf("not supported")` .


And the old `log/buffer.go` is difficult to use, developers have to
write a lot of `Contains` and `Sleep` code.


The new `LogChecker` is designed to be a general approach to help to
assert some messages appearing or not appearing in logs.
2023-04-21 16:32:25 -04:00
silverwind 938b591994
Remove most path-based golangci exclusions (#24214)
They are non-obvious and do not survive refactor.

Will replace with `//nolint` comments after CI results are in.
2023-04-19 22:08:01 -04:00
yp05327 fa3495183b
Add migration to fix external unit access mode of owner/admin team (#24117)
Fix the incorrect migration in #23675 and #24012

External Unit (Tracker and Wiki) access mode should be `read` in
owner/admin team.
2023-04-15 09:52:44 -04:00
yp05327 b7221bec34
Fix admin team access mode value in team_unit table (#24012)
Same as https://github.com/go-gitea/gitea/pull/23675
Feedback:
https://github.com/go-gitea/gitea/pull/23879#issuecomment-1500923636
2023-04-13 21:06:10 +02:00
yp05327 6eb856c614
Fix owner team access mode value in team_unit table (#23675)
All `access_mode` value of Owner Teams are 0(AccessModeNone) in
`team_unit` table, which should be 4(AccessModeOwner)
In `team` table:

![image](https://user-images.githubusercontent.com/18380374/227409457-1b9660ae-8cf7-49c8-a013-1850b46baebc.png)
In `team_unit` table:

![image](https://user-images.githubusercontent.com/18380374/227409429-a793dd90-4ae1-4191-b95b-e288c591f9fd.png)

ps: In https://github.com/go-gitea/gitea/pull/23630, `access_mode` in
`team_unit` is used to check the team unit permission, but I found that
user can not see issues in owned org repos.
2023-04-03 10:36:35 -04:00
KN4CK3R fbd4eaceed
Display image size for multiarch container images (#23821)
Fixes #23771

Changes the display of different architectures for multiarch images to
show the image size:

![grafik](https://user-images.githubusercontent.com/1666336/228781477-cc76c4d1-4728-434f-8a27-fc008790d924.png)
2023-04-02 17:53:37 +08:00
Zettat123 b958dba1a0
Improve indices for action table (#23532)
Close #21611 

Add the index mentioned in
https://github.com/go-gitea/gitea/issues/21611#issuecomment-1451113252 .
Since we already have an index for `("created_unix", "user_id",
"is_deleted")` columns on PostgreSQL, I removed the database type check
to apply this index to all types of databases.
2023-03-24 23:44:33 +08:00
Jason Song 9f39def991
Drop migration for ForeignReference (#23605)
Fix
https://github.com/go-gitea/gitea/issues/21086#issuecomment-1476560381

Related to #21721
2023-03-23 15:01:55 -04:00
sillyguodong 371520d7ab
Display the version of runner in the runner list (#23490)
Close: #23489 

### Change
1. Add version column to action_runner table.
2. Read the runner version from the request header, and update it in DB.
3. Display version in runner list

### Screenshot

![image](https://user-images.githubusercontent.com/33891828/225220990-98bc0158-4403-4e6c-9805-31bbbc65a802.png)
2023-03-19 22:19:40 -04:00
yp05327 8e45fcb63a
Do not store user projects as organization projects (#23353)
A part of https://github.com/go-gitea/gitea/pull/22865

At first, I think we do not need 3 ProjectTypes, as we can check user
type, but it seems that it is not database friendly.

---------

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
2023-03-17 21:07:23 +08:00
Lunny Xiao 27494ed20d
Fix missed migration in #22235 (#23482)
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-03-15 10:33:10 +01:00
Lunny Xiao c492e86312
Remove wrongly added column on migration test fixtures (#23456)
Fix https://drone.gitea.io/go-gitea/gitea/69418/3/8
Migration fixtures are in `models/migrations/fixtures`, every folder
will be used only by the test with the same name.
For `Test_DeleteOrphanedIssueLabels`, the fixture should keep consistent
as the database structure at that time. So the newly added `exclusive`
is not right. Just revert the change in
https://github.com/go-gitea/gitea/pull/22585/files#diff-f8db9cbbaa10bf7b27eb726884454db821a4b4f8cb9a0d50435555908761bbcb
2023-03-14 19:49:59 +08:00
KN4CK3R 2173f14708
Add user webhooks (#21563)
Currently we can add webhooks for organizations but not for users. This
PR adds the latter. You can access it from the current users settings.


![grafik](https://user-images.githubusercontent.com/1666336/197391408-15dfdc23-b476-4d0c-82f7-9bc9b065988f.png)
2023-03-10 08:28:32 -06:00
Jason Song c84238800b
Refactor setting.Database.UseXXX to methods (#23354)
Replace #23350.

Refactor `setting.Database.UseMySQL` to
`setting.Database.Type.IsMySQL()`.

To avoid mismatching between `Type` and `UseXXX`.

This refactor can fix the bug mentioned in #23350, so it should be
backported.
2023-03-07 18:51:06 +08:00
Jason Song edf98a2dc3
Require approval to run actions for fork pull request (#22803)
Currently, Gitea will run actions automatically which are triggered by
fork pull request. It's a security risk, people can create a PR and
modify the workflow yamls to execute a malicious script.

So we should require approval for first-time contributors, which is the
default strategy of a public repo on GitHub, see [Approving workflow
runs from public
forks](https://docs.github.com/en/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks).

Current strategy:

- don't need approval if it's not a fork PR;
- always need approval if the user is restricted;
- don't need approval if the user can write;
- don't need approval if the user has been approved before;
- otherwise, need approval.

https://user-images.githubusercontent.com/9418365/217207121-badf50a8-826c-4425-bef1-d82d1979bc81.mov

GitHub has an option for that, you can see that at
`/<owner>/<repo>/settings/actions`, and we can support that later.

<img width="835" alt="image"
src="https://user-images.githubusercontent.com/9418365/217199990-2967e68b-e693-4e59-8186-ab33a1314a16.png">

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-24 15:58:49 +08:00
zeripath 1319ba6742
Use minio/sha256-simd for accelerated SHA256 (#23052)
minio/sha256-simd provides additional acceleration for SHA256 using
AVX512, SHA Extensions for x86 and ARM64 for ARM.

It provides a drop-in replacement for crypto/sha256 and if the
extensions are not available it falls back to standard crypto/sha256.

---------

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2023-02-22 14:21:46 -05:00
Yarden Shoham 6840a8ccfc
Add comment marking the end of database migrations in 1.19.0 (#22975)
There will be no more migrations in `1.19.0`

---------

Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
2023-02-20 10:30:36 +08:00
Lunny Xiao c53ad052d8
Refactor the setting to make unit test easier (#22405)
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.

- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.

Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?

---------

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
2023-02-20 00:12:01 +08:00
zeripath 2b02343e21
Migration v244.go should be v243.go (#22988) 2023-02-19 09:24:23 -05:00
Brecht Van Lommel 6221a6fd54
Scoped labels (#22585)
Add a new "exclusive" option per label. This makes it so that when the
label is named `scope/name`, no other label with the same `scope/`
prefix can be set on an issue.

The scope is determined by the last occurence of `/`, so for example
`scope/alpha/name` and `scope/beta/name` are considered to be in
different scopes and can coexist.

Exclusive scopes are not enforced by any database rules, however they
are enforced when editing labels at the models level, automatically
removing any existing labels in the same scope when either attaching a
new label or replacing all labels.

In menus use a circle instead of checkbox to indicate they function as
radio buttons per scope. Issue filtering by label ensures that only a
single scoped label is selected at a time. Clicking with alt key can be
used to remove a scoped label, both when editing individual issues and
batch editing.

Label rendering refactor for consistency and code simplification:

* Labels now consistently have the same shape, emojis and tooltips
everywhere. This includes the label list and label assignment menus.
* In label list, show description below label same as label menus.
* Don't use exactly black/white text colors to look a bit nicer.
* Simplify text color computation. There is no point computing luminance
in linear color space, as this is a perceptual problem and sRGB is
closer to perceptually linear.
* Increase height of label assignment menus to show more labels. Showing
only 3-4 labels at a time leads to a lot of scrolling.
* Render all labels with a new RenderLabel template helper function.

Label creation and editing in multiline modal menu:

* Change label creation to open a modal menu like label editing.
* Change menu layout to place name, description and colors on separate
lines.
* Don't color cancel button red in label editing modal menu.
* Align text to the left in model menu for better readability and
consistent with settings layout elsewhere.

Custom exclusive scoped label rendering:

* Display scoped label prefix and suffix with slightly darker and
lighter background color respectively, and a slanted edge between them
similar to the `/` symbol.
* In menus exclusive labels are grouped with a divider line.

---------

Co-authored-by: Yarden Shoham <hrsi88@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2023-02-18 21:17:39 +02:00
zeripath 52dd383b6d
Increase Content field size of gpg_key_import to MEDIUMTEXT (#22897)
Unfortunately #20896 does not completely prevent Data too long issues
and GPGKeyImport needs to be increased too.

Fix #22896

Signed-off-by: Andrew Thornton <art27@cantab.net>
2023-02-16 12:08:40 -06:00
Nathaniel Sabanski d4a9b35c4b
Fix migration issue. (#22867)
See:
https://github.com/go-gitea/gitea/pull/22112#issuecomment-1426872992
2023-02-11 21:28:41 +00:00
Nathaniel Sabanski fb1a2a13f0
Preview images for Issue cards in Project Board view (#22112)
Original Issue: https://github.com/go-gitea/gitea/issues/22102

This addition would be a big benefit for design and art teams using the
issue tracking.

The preview will be the latest "image type" attachments on an issue-
simple, and allows for automatic updates of the cover image as issue
progress is made!

This would make Gitea competitive with Trello... wouldn't it be amazing
to say goodbye to Atlassian products? Ha.

First image is the most recent, the SQL will fetch up to 5 latest images
(URL string).

All images supported by browsers plus upcoming formats: *.avif *.bmp
*.gif *.jpg *.jpeg *.jxl *.png *.svg *.webp

The CSS will try to center-align images until it cannot, then it will
left align with overflow hidden. Single images get to be slightly
larger!

Tested so far on: Chrome, Firefox, Android Chrome, Android Firefox.

Current revision with light and dark themes:

![image](https://user-images.githubusercontent.com/24665/207066878-58e6bf73-0c93-4caa-8d40-38f4432b3578.png)


![image](https://user-images.githubusercontent.com/24665/207066555-293f65c3-e706-4888-8516-de8ec632d638.png)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
2023-02-11 16:12:41 +08:00
Jason Song 4011821c94
Implement actions (#21937)
Close #13539.

Co-authored by: @lunny @appleboy @fuxiaohei and others.

Related projects:
- https://gitea.com/gitea/actions-proto-def
- https://gitea.com/gitea/actions-proto-go
- https://gitea.com/gitea/act
- https://gitea.com/gitea/act_runner

### Summary

The target of this PR is to bring a basic implementation of "Actions",
an internal CI/CD system of Gitea. That means even though it has been
merged, the state of the feature is **EXPERIMENTAL**, and please note
that:

- It is disabled by default;
- It shouldn't be used in a production environment currently;
- It shouldn't be used in a public Gitea instance currently;
- Breaking changes may be made before it's stable.

**Please comment on #13539 if you have any different product design
ideas**, all decisions reached there will be adopted here. But in this
PR, we don't talk about **naming, feature-creep or alternatives**.

### ⚠️ Breaking

`gitea-actions` will become a reserved user name. If a user with the
name already exists in the database, it is recommended to rename it.

### Some important reviews

- What is `DEFAULT_ACTIONS_URL` in `app.ini` for?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1055954954
- Why the api for runners is not under the normal `/api/v1` prefix?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061173592
- Why DBFS?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061301178
- Why ignore events triggered by `gitea-actions` bot?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1063254103
- Why there's no permission control for actions?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1090229868

### What it looks like

<details>

#### Manage runners

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205870657-c72f590e-2e08-4cd4-be7f-2e0abb299bbf.png">

#### List runs

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205872794-50fde990-2b45-48c1-a178-908e4ec5b627.png">


#### View logs

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205872501-9b7b9000-9542-4991-8f55-18ccdada77c3.png">



</details>

### How to try it

<details>

#### 1. Start Gitea

Clone this branch and [install from
source](https://docs.gitea.io/en-us/install-from-source).

Add additional configurations in `app.ini` to enable Actions:

```ini
[actions]
ENABLED = true
```

Start it.

If all is well, you'll see the management page of runners:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205877365-8e30a780-9b10-4154-b3e8-ee6c3cb35a59.png">


#### 2. Start runner

Clone the [act_runner](https://gitea.com/gitea/act_runner), and follow
the
[README](https://gitea.com/gitea/act_runner/src/branch/main/README.md)
to start it.

If all is well, you'll see a new runner has been added:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205878000-216f5937-e696-470d-b66c-8473987d91c3.png">

#### 3. Enable actions for a repo

Create a new repo or open an existing one, check the `Actions` checkbox
in settings and submit.

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205879705-53e09208-73c0-4b3e-a123-2dcf9aba4b9c.png">
<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205879383-23f3d08f-1a85-41dd-a8b3-54e2ee6453e8.png">

If all is well, you'll see a new tab "Actions":

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205881648-a8072d8c-5803-4d76-b8a8-9b2fb49516c1.png">

#### 4. Upload workflow files

Upload some workflow files to `.gitea/workflows/xxx.yaml`, you can
follow the [quickstart](https://docs.github.com/en/actions/quickstart)
of GitHub Actions. Yes, Gitea Actions is compatible with GitHub Actions
in most cases, you can use the same demo:

```yaml
name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
      - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: List files in the repository
        run: |
          ls ${{ github.workspace }}
      - run: echo "🍏 This job's status is ${{ job.status }}."
```

If all is well, you'll see a new run in `Actions` tab:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205884473-79a874bc-171b-4aaf-acd5-0241a45c3b53.png">

#### 5. Check the logs of jobs

Click a run and you'll see the logs:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205884800-994b0374-67f7-48ff-be9a-4c53f3141547.png">

#### 6. Go on

You can try more examples in [the
documents](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)
of GitHub Actions, then you might find a lot of bugs.

Come on, PRs are welcome.

</details>

See also: [Feature Preview: Gitea
Actions](https://blog.gitea.io/2022/12/feature-preview-gitea-actions/)

---------

Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: ChristopherHX <christopher.homberger@web.de>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2023-01-31 09:45:19 +08:00
Chongyi Zheng de484e86bc
Support scoped access tokens (#20908)
This PR adds the support for scopes of access tokens, mimicking the
design of GitHub OAuth scopes.

The changes of the core logic are in `models/auth` that `AccessToken`
struct will have a `Scope` field. The normalized (no duplication of
scope), comma-separated scope string will be stored in `access_token`
table in the database.
In `services/auth`, the scope will be stored in context, which will be
used by `reqToken` middleware in API calls. Only OAuth2 tokens will have
granular token scopes, while others like BasicAuth will default to scope
`all`.
A large amount of work happens in `routers/api/v1/api.go` and the
corresponding `tests/integration` tests, that is adding necessary scopes
to each of the API calls as they fit.


- [x] Add `Scope` field to `AccessToken`
- [x] Add access control to all API endpoints
- [x] Update frontend & backend for when creating tokens
- [x] Add a database migration for `scope` column (enable 'all' access
to past tokens)

I'm aiming to complete it before Gitea 1.19 release.

Fixes #4300
2023-01-17 15:46:03 -06:00
zeripath 2cc3a6381c
Add cron method to gc LFS MetaObjects (#22385)
This PR adds a task to the cron service to allow garbage collection of
LFS meta objects. As repositories may have a large number of
LFSMetaObjects, an updated column is added to this table and it is used
to perform a generational GC to attempt to reduce the amount of work.
(There may need to be a bit more work here but this is probably enough
for the moment.)

Fix #7045

Signed-off-by: Andrew Thornton <art27@cantab.net>
2023-01-16 13:50:53 -06:00
delvh 0f4e1b9ac6
Restructure webhook module (#22256)
Previously, there was an `import services/webhooks` inside
`modules/notification/webhook`.
This import was removed (after fighting against many import cycles).
Additionally, `modules/notification/webhook` was moved to
`modules/webhook`,
and a few structs/constants were extracted from `models/webhooks` to
`modules/webhook`.

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-01-01 23:23:15 +08:00
Jason Song 71ca3067bc
Check primary keys for all tables and drop ForeignReference (#21721)
Some dbs require that all tables have primary keys, see
- #16802
- #21086

We can add a test to keep it from being broken again.

Edit:

~Added missing primary key for `ForeignReference`~ Dropped the
`ForeignReference` table to satisfy the check, so it closes #21086.

More context can be found in comments.

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
2022-12-23 19:35:43 +08:00
Jason Song 659055138b
Secrets storage with SecretKey encrypted (#22142)
Fork of #14483, but [gave up
MasterKey](https://github.com/go-gitea/gitea/pull/14483#issuecomment-1350728557),
and fixed some problems.

Close #12065.
Needed by #13539.

Featrues:
- Secrets for repo and org, not user yet.
- Use SecretKey to encrypte/encrypt secrets.
- Trim spaces of secret value.
- Add a new locale ini block, to make it easy to support secrets for
user.

Snapshots:

Repo level secrets:

![image](https://user-images.githubusercontent.com/9418365/207823319-b8a4903f-38ca-4af7-9d05-336a5af906f3.png)

Rrg level secrets

![image](https://user-images.githubusercontent.com/9418365/207823371-8bd02e93-1928-40d1-8c76-f48b255ace36.png)

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2022-12-20 17:07:13 +08:00
KN4CK3R 3c59d31bc6
Add API management for issue/pull and comment attachments (#21783)
Close #14601
Fix #3690

Revive of #14601.
Updated to current code, cleanup and added more read/write checks.

Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andre Bruch <ab@andrebruch.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Norwin <git@nroo.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-12-09 14:35:56 +08:00
silverwind 0585ac3ac6
Update go dev dependencies (#22064)
`golangci-lint`
[deprecated](https://github.com/golangci/golangci-lint/issues/1841) a
bunch of linters, removed them.
2022-12-08 16:21:37 +08:00
Jason Song f59a74852b
Update gitea-vet to check FSFE REUSE (#22004)
Related to:
- #21840
- https://gitea.com/gitea/gitea-vet/pulls/21

What it looks like when it's working:
https://drone.gitea.io/go-gitea/gitea/64040/1/5

All available SPDX license identifiers: [SPDX License
List](https://spdx.org/licenses/).

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-12-02 22:14:57 +08:00
Jason Song 9607750b5e
Replace fmt.Sprintf with hex.EncodeToString (#21960)
`hex.EncodeToString` has better performance than `fmt.Sprintf("%x",
[]byte)`, we should use it as much as possible.

I'm not an extreme fan of performance, so I think there are some
exceptions:

- `fmt.Sprintf("%x", func(...)[N]byte())`
- We can't slice the function return value directly, and it's not worth
adding lines.
    ```diff
    func A()[20]byte { ... }
    - a := fmt.Sprintf("%x", A())
    - a := hex.EncodeToString(A()[:]) // invalid
    + tmp := A()
    + a := hex.EncodeToString(tmp[:])
    ```
- `fmt.Sprintf("%X", []byte)`
- `strings.ToUpper(hex.EncodeToString(bytes))` has even worse
performance.
2022-11-28 11:19:18 +00:00
flynnnnnnnnnn e81ccc406b
Implement FSFE REUSE for golang files (#21840)
Change all license headers to comply with REUSE specification.

Fix #16132

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
Jason Song 702f4b521e
Add index for access_token (#21908)
The `token_last_eight` field has been used in `GetAccessTokenBySHA `:
2022-11-23 20:49:41 -06:00
zeripath 4c00d8f916
Move migration test fixtures to the correct directories (#21901)
Unfortunately #21549 changed the name of Testcases without changing
their associated fixture directories.

Fix #21854

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-23 18:24:55 +08:00
KN4CK3R 32db62515f
Add package registry cleanup rules (#21658)
Fixes #20514
Fixes #20766
Fixes #20631

This PR adds Cleanup Rules for the package registry. This allows to
delete unneeded packages automatically. Cleanup rules can be set up from
the user or org settings.
Please have a look at the documentation because I'm not a native english
speaker.

Rule Form

![grafik](https://user-images.githubusercontent.com/1666336/199330792-c13918a6-e196-4e71-9f53-18554515edca.png)

Rule List

![grafik](https://user-images.githubusercontent.com/1666336/199331261-5f6878e8-a80c-4985-800d-ebb3524b1a8d.png)

Rule Preview

![grafik](https://user-images.githubusercontent.com/1666336/199330917-c95e4017-cf64-4142-a3e4-af18c4f127c3.png)

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-20 16:08:38 +02:00
wxiaoguang 92525ddffd
Init git module before database migration (#21764)
Close #21761

Some database migrations depend on the git module.
2022-11-10 14:22:39 +00:00
Lunny Xiao 91c7a3e66f
Fix tests on migrations (#21705) 2022-11-08 12:07:46 +08:00
oliverpool b6e81357bd
Add Webhook authorization header (#20926)
_This is a different approach to #20267, I took the liberty of adapting
some parts, see below_

## Context

In some cases, a weebhook endpoint requires some kind of authentication.
The usual way is by sending a static `Authorization` header, with a
given token. For instance:

- Matrix expects a `Bearer <token>` (already implemented, by storing the
header cleartext in the metadata - which is buggy on retry #19872)
- TeamCity #18667
- Gitea instances #20267
- SourceHut https://man.sr.ht/graphql.md#authentication-strategies (this
is my actual personal need :)

## Proposed solution

Add a dedicated encrypt column to the webhook table (instead of storing
it as meta as proposed in #20267), so that it gets available for all
present and future hook types (especially the custom ones #19307).

This would also solve the buggy matrix retry #19872.

As a first step, I would recommend focusing on the backend logic and
improve the frontend at a later stage. For now the UI is a simple
`Authorization` field (which could be later customized with `Bearer` and
`Basic` switches):


![2022-08-23-142911](https://user-images.githubusercontent.com/3864879/186162483-5b721504-eef5-4932-812e-eb96a68494cc.png)

The header name is hard-coded, since I couldn't fine any usecase
justifying otherwise.

## Questions

- What do you think of this approach? @justusbunsi @Gusted @silverwind 
- ~~How are the migrations generated? Do I have to manually create a new
file, or is there a command for that?~~
- ~~I started adding it to the API: should I complete it or should I
drop it? (I don't know how much the API is actually used)~~

## Done as well:

- add a migration for the existing matrix webhooks and remove the
`Authorization` logic there


_Closes #19872_

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: delvh <dev.lh@web.de>
2022-11-03 20:23:20 +02:00
KN4CK3R f8f859b8fa
Alter package_version.metadata_json to LONGTEXT (#21667) 2022-11-03 15:28:46 +08:00
Lunny Xiao e72acd5e5b
Split migrations folder (#21549)
There are too many files in `models/migrations` folder so that I split
them into sub folders.
2022-11-02 16:54:36 +08:00
Lunny Xiao f337c32e86
Add index for hook_task table (#21545)
Since `hook_id` and `uuid` will become a search condition column. It's
better to add some index for them.
2022-10-28 13:05:39 +02:00
delvh 0ebb45cfe7
Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)
Found using
`find . -type f -name '*.go' -print -exec vim {} -c
':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;`

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-24 20:29:17 +01:00
M Hickford 191a74d622
Record OAuth client type at registration (#21316)
The OAuth spec [defines two types of
client](https://datatracker.ietf.org/doc/html/rfc6749#section-2.1),
confidential and public. Previously Gitea assumed all clients to be
confidential.

> OAuth defines two client types, based on their ability to authenticate
securely with the authorization server (i.e., ability to
>   maintain the confidentiality of their client credentials):
>
>   confidential
> Clients capable of maintaining the confidentiality of their
credentials (e.g., client implemented on a secure server with
> restricted access to the client credentials), or capable of secure
client authentication using other means.
>
>   **public
> Clients incapable of maintaining the confidentiality of their
credentials (e.g., clients executing on the device used by the resource
owner, such as an installed native application or a web browser-based
application), and incapable of secure client authentication via any
other means.**
>
> The client type designation is based on the authorization server's
definition of secure authentication and its acceptable exposure levels
of client credentials. The authorization server SHOULD NOT make
assumptions about the client type.

 https://datatracker.ietf.org/doc/html/rfc8252#section-8.4

> Authorization servers MUST record the client type in the client
registration details in order to identify and process requests
accordingly.

Require PKCE for public clients:
https://datatracker.ietf.org/doc/html/rfc8252#section-8.1

> Authorization servers SHOULD reject authorization requests from native
apps that don't use PKCE by returning an error message

Fixes #21299

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-10-24 15:59:24 +08:00
wxiaoguang dcd9fc7ee8
Refactor git command arguments and make all arguments to be safe to be used (#21535)
Follow #21464

Make all git command arguments strictly safe. Most changes are one-to-one replacing, keep all existing logic.
2022-10-23 22:44:45 +08:00
Ashley Nelson 82ecd3b19e
Update milestone counters when issue is deleted (#21459)
When actions besides "delete" are performed on issues, the milestone
counter is updated. However, since deleting issues goes through a
different code path, the associated milestone's count wasn't being
updated, resulting in inaccurate counts until another issue in the same
milestone had a non-delete action performed on it.

I verified this change fixes the inaccurate counts using a local docker
build.

Fixes #21254

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-10-22 23:08:10 +08:00
KN4CK3R c3b2e44392
Add team member invite by email (#20307)
Allows to add (not registered) team members by email.

related #5353

Invite by mail:

![grafik](https://user-images.githubusercontent.com/1666336/178154779-adcc547f-c0b7-4a2a-a131-4e41a3d9d3ad.png)

Pending invitations:

![grafik](https://user-images.githubusercontent.com/1666336/178154882-9d739bb8-2b04-46c1-a025-c1f4be26af98.png)

Email:

![grafik](https://user-images.githubusercontent.com/1666336/178164716-f2f90893-7ba6-4a5e-a3db-42538a660258.png)

Join form:

![grafik](https://user-images.githubusercontent.com/1666336/178154840-aaab983a-d922-4414-b01a-9b1a19c5cef7.png)

Co-authored-by: Jack Hay <jjphay@gmail.com>
2022-10-19 14:40:28 +02:00
Lunny Xiao f860a6d2e4
Add system setting table with cache and also add cache supports for user setting (#18058) 2022-10-17 07:29:26 +08:00
KN4CK3R 0e57ff7eee
Add generic set type (#21408)
This PR adds a generic set type to get rid of maps used as sets.

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-12 13:18:26 +08:00
KN4CK3R 30ca91666e
Set SemverCompatible to false for Conan packages (#21275)
Fixes #21250
Related #20414

Conan packages don't have to follow SemVer.
The migration fixes the setting for all existing Conan and Generic
(#20414) packages.
2022-10-07 12:22:05 +08:00
zeripath 58a80ba69c
Add another index for Action table on postgres (#21033)
In #21031 we have discovered that on very big tables postgres will use a
search involving the sort term in preference to the restrictive index.

Therefore we add another index for postgres and update the original migration.

Fix #21031

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-09-03 17:27:59 +01:00
Kyle D c8ded77680
Kd/ci playwright go test (#20123)
* Add initial playwright config

* Simplify Makefile

* Simplify Makefile

* Use correct config files

* Update playwright settings

* Fix package-lock file

* Don't use test logger for e2e tests

* fix frontend lint

* Allow passing TEST_LOGGER variable

* Init postgres database

* use standard gitea env variables

* Update playwright

* update drone

* Move empty env var to commands

* Cleanup

* Move integrations to subfolder

* tests integrations to tests integraton

* Run e2e tests with go test

* Fix linting

* install CI deps

* Add files to ESlint

* Fix drone typo

* Don't log to console in CI

* Use go test http server

* Add build step before tests

* Move shared init function to common package

* fix drone

* Clean up tests

* Fix linting

* Better mocking for page + version string

* Cleanup test generation

* Remove dependency on gitea binary

* Fix linting

* add initial support for running specific tests

* Add ACCEPT_VISUAL variable

* don't require git-lfs

* Add initial documentation

* Review feedback

* Add logged in session test

* Attempt fixing drone race

* Cleanup and bump version

* Bump deps

* Review feedback

* simplify installation

* Fix ci

* Update install docs
2022-09-02 15:18:23 -04:00
zeripath 9d6a2034bc
Increase Content field size of gpg_key and public_key to MEDIUMTEXT (#20896)
Unfortunately some keys are too big to fix within the 65535 limit of TEXT on MySQL
this causes issues with these large keys.

Therefore increase these fields to MEDIUMTEXT.

Fix #20894

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-08-22 14:32:28 +01:00
Lunny Xiao 5d0f643461
Fix migration file name (#20843) 2022-08-18 13:38:59 +08:00
techknowlogick d8e6c99125
Add badge capabilities to users (#20607)
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-08-18 02:25:25 +03:00
wxiaoguang 75d96f4a02
Refactor legacy git init (#20376)
* merge `CheckLFSVersion` into `InitFull` (renamed from `InitWithSyncOnce`)
* remove the `Once` during git init, no data-race now
* for doctor sub-commands, `InitFull` should only be called in initialization stage

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-08-09 11:22:24 +08:00
Lunny Xiao 5b4a84beb4
Fix v220 migration to be compatible for MSSQL 2008 r2 (#20702) 2022-08-08 02:16:22 +02:00
zeripath e819da0837
WebAuthn CredentialID field needs to be increased in size (#20530)
WebAuthn have updated their specification to set the maximum size of the
CredentialID to 1023 bytes. This is somewhat larger than our current
size and therefore we need to migrate.

The PR changes the struct to add CredentialIDBytes and migrates the CredentialID string 
to the bytes field before another migration drops the old CredentialID field. Another migration
renames this field back.

Fix #20457

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-07-30 15:25:26 +02:00
KN4CK3R 86e5268c39
Add Docker /v2/_catalog endpoint (#20469)
* Added properties for packages.
* Fixed authenticate header format.
* Added _catalog endpoint.
* Check owner visibility.
* Extracted condition.
* Added test for _catalog.

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-07-28 11:59:39 +08:00
silverwind 7d20c8323a
Fix commit status icon when in subdirectory (#20285)
When viewing a subdirectory and the latest commit to that directory in
the table, the commit status icon incorrectly showed the status of the
HEAD commit instead of the latest for that directory.
2022-07-15 14:01:32 +01:00
Chongyi Zheng 49f9d43afe
Implement sync push mirror on commit (#19411)
Support synchronizing with the push mirrors whenever new commits are pushed or synced from pull mirror.

Related Issues: #18220

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-07-08 20:45:12 +01:00
zeripath 5c9c0b8c1e
Refix indices on actions table (#20158)
Unforunately the previous PR #20035 created indices that were not helpful
for SQLite. This PR adjusts these after testing using the try.gitea.io db.

Fix #20129

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-07-01 17:04:01 +01:00
Gusted 711cbcce8d
Use neutral language in comments and docs (#20135)
- Replace `his/her` to `their`, as it's more neutral language.
2022-06-25 17:50:12 -05:00
Wim cb50375e2b
Add more linters to improve code readability (#19989)
Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability

- nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length.
- unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions
- wastedassign - https://github.com/sanposhiho/wastedassign -  wastedassign finds wasted assignment statements.
- notlintlint -  Reports ill-formed or insufficient nolint directives
- stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent
  - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
2022-06-20 12:02:49 +02:00
Wim 62104b4896
Alter hook_task TEXT fields to LONGTEXT (#20038)
Mysql TEXT has a limit of 64KB, change this to LONGTEXT in mysql only so we can have bigger hook payloads.

Postgresql has unlimited TEXT - https://www.postgresql.org/docs/current/datatype-character.html
Sqlite has unlimited TEXT - https://www.sqlitetutorial.net/sqlite-data-types/#:~:text=The%20maximum%20length%20of%20TEXT,SQLite%20supports%20various%20character%20encodings.

Same issue as #16656 but for hook_task

Fixes #10252, #19679, #3561
2022-06-19 19:47:04 +01:00