1
0
Fork 0
mirror of https://codeberg.org/forgejo/forgejo.git synced 2024-12-29 20:13:27 +00:00
Commit graph

18 commits

Author SHA1 Message Date
Lunny Xiao d6dd6d641b
Fix all possible setting error related storages and added some tests ()
Follow up 

Fix  

This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.

- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-06-14 11:42:38 +08:00
FuXiaoHei 61ad4c607b
fix minio storage iterator path ()
minio storage iterator shows different behavior with local fs iterator.

in local fs storage:

``` go
s.IterateObjects("prefix", func(path,obj)
     println(path) // show "prefix/xxx.file"
})
```

in minio storage:

```go
s.IterateObjects("prefix", func(path,obj)
     println(path) // show "xxx.file"
})
```

I think local fs is correct, minio use wrong `basePath` to trim storage
path prefix.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-13 22:33:25 +00:00
wxiaoguang 5727056ea1
Make minio package support legacy MD5 checksum ()
A feedback from discord:
https://discord.com/channels/322538954119184384/561007778139734027/1090185427115319386

Some storages like:

 * https://developers.cloudflare.com/r2/api/s3/api/
 * https://www.backblaze.com/b2/docs/s3_compatible_api.html

They do not support "x-amz-checksum-algorithm" header

But minio recently uses that header with CRC32C by default. So we have
to tell minio to use legacy MD5 checksum.

I guess this needs to be backported because IIRC we 1.19 and 1.20 are
using similar minio package.


The minio package code for SendContentMD5 looks like this:

<details>

<img width="755" alt="image"
src="https://user-images.githubusercontent.com/2114189/228186768-4f2f6f67-62b9-4aee-9251-5af714ad9674.png">

</details>
2023-03-28 11:10:24 -04:00
wxiaoguang ce9dee5a1e
Introduce path Clean/Join helper functions ()
Since  has conflicts with latest commits, this PR is my proposal
for fixing 

Details are in the comments

And refactor the `modules/options` module, to make it always use
"filepath" to access local files.

Benefits:

* No need to do `util.CleanPath(strings.ReplaceAll(p, "\\", "/"))),
"/")` any more (not only one before)
* The function behaviors are clearly defined
2023-03-21 16:02:49 -04:00
FuXiaoHei cdc9e91750
add path prefix to ObjectStorage.Iterator ()
Support to iterator subdirectory in ObjectStorage for
ObjectStorage.Iterator method.

It's required for https://github.com/go-gitea/gitea/pull/22738 to make
artifact files cleanable.

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-03-13 18:23:51 +08:00
Lunny Xiao b116418f05
Use CleanPath instead of path.Clean ()
As title.
2023-03-08 20:17:39 +08:00
Yarden Shoham 0e7bec1849
Add InsecureSkipVerify to Minio Client for Storage ()
Allows using Minio with untrusted certificates

Closes 

Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
2023-02-27 16:26:13 +00:00
flynnnnnnnnnn e81ccc406b
Implement FSFE REUSE for golang files ()
Change all license headers to comply with REUSE specification.

Fix 

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
zeripath 3f71ab9a12
Clean paths when looking in Storage ()
* Clean paths when looking in Storage

Ensure paths are clean for minio aswell as local storage.

Use url.Path not RequestURI/EscapedPath in storageHandler.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Apply suggestions from code review

Co-authored-by: Lauris BH <lauris@nix.lv>
2022-03-22 17:02:26 -04:00
6543 54e9ee37a7
format with gofumpt ()
* gofumpt -w -l .

* gofumpt -w -l -extra .

* Add linter

* manual fix

* change make fmt
2022-01-20 18:46:10 +01:00
Lunny Xiao a807031a30
Fix storage Iterate bug and Add storage doctor to delete garbage attachments ()
* Fix storage Iterate bug and Add storage doctor to delete garbage attachments

* Close object when used
2021-09-06 16:46:20 +02:00
zeripath 0bb8bd8190
Add size to Save function ()
Fix 

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-03 18:19:59 +02:00
zeripath 25b7766673
When handling errors in storageHandler check underlying error ()
Unfortunately there was a mistake in  which fails to handle
os.PathError wrapping an os.ErrNotExist

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-17 21:29:06 -04:00
zeripath 91f2afdb54
Prevent panics with missing storage ()
* The `.Use` of storageHandler before setting up the template renderer
causes a panic if there is an error to log.
* The error passed to `ctx.Error` in that case may contain sensitive
information and should not be rendered to the end user. We should
instead log the error and render a simple error message.
* There is no handling of missing avatars and this needs a 404. Minio
errors need to be mapped to standard golang errors such as
os.ErrNotExist.
* There is no logging when storage is set up.

Related 

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-10-15 23:51:06 -04:00
zeripath 6b1266b6b3
Provide self-registering storage system ()
* Provide self-registering storage system

Signed-off-by: Andrew Thornton <art27@cantab.net>

* More simplification

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Remove old strings from setting

Signed-off-by: Andrew Thornton <art27@cantab.net>

* oops attachments not attachment

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-10-12 23:58:34 -04:00
Lunny Xiao 3878e985b6
Add default storage configurations ()
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
2020-09-29 12:05:13 +03:00
Lunny Xiao 7a5465fc56
LFS support to be stored on minio ()
* LFS support to be stored on minio

* Fix test

* Fix lint

* Fix lint

* Fix check

* Fix test

* Update documents and add migration for LFS

* Fix some bugs
2020-09-08 23:45:10 +08:00
Lunny Xiao 62e6c9bc6c
Add a storage layer for attachments ()
* Add a storage layer for attachments

* Fix some bug

* fix test

* Fix copyright head and lint

* Fix bug

* Add setting for minio and flags for migrate-storage

* Add documents

* fix lint

* Add test for minio store type on attachments

* fix test

* fix test

* Apply suggestions from code review

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Add warning when storage migrated successfully

* Fix drone

* fix test

* rebase

* Fix test

* display the error on console

* Move minio test to amd64 since minio docker don't support arm64

* refactor the codes

* add trace

* Fix test

* remove log on xorm

* Fi download bug

* Add a storage layer for attachments

* Add setting for minio and flags for migrate-storage

* fix lint

* Add test for minio store type on attachments

* Apply suggestions from code review

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Fix drone

* fix test

* Fix test

* display the error on console

* Move minio test to amd64 since minio docker don't support arm64

* refactor the codes

* add trace

* Fix test

* Add URL function to serve attachments directly from S3/Minio

* Add ability to enable/disable redirection in attachment configuration

* Fix typo

* Add a storage layer for attachments

* Add setting for minio and flags for migrate-storage

* fix lint

* Add test for minio store type on attachments

* Apply suggestions from code review

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Fix drone

* fix test

* Fix test

* display the error on console

* Move minio test to amd64 since minio docker don't support arm64

* don't change unrelated files

* Fix lint

* Fix build

* update go.mod and go.sum

* Use github.com/minio/minio-go/v6

* Remove unused function

* Upgrade minio to v7 and some other improvements

* fix lint

* Fix go mod

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Tyler <tystuyfzand@gmail.com>
2020-08-18 12:23:45 +08:00