Continues on from #19202.
Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them.
This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile.
The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines.
If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process.
The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound.
In this way we should be able to more easily examine goroutines that have been stuck.
A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move keys to models/keys
* Rename models/keys -> models/asymkey
* change the missed package name
* Fix package alias
* Fix test
* Fix docs
* Fix test
* Fix test
* merge
* Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH
There is a subtle bug in the SSH library x/crypto/ssh which makes the incorrect
assumption that the public key type is the same as the signature algorithm type.
This means that only ssh-rsa signatures are offered by default.
This PR adds a workaround around this problem.
Fix#17175
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per review
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Second attempt at preventing zombies
* Ensure that the pipes are closed in ssh.go
* Ensure that a cancellable context is passed up in cmd/* http requests
* Make cmd.fail return properly so defers are obeyed
* Ensure that something is sent to stdout in case of blocks here
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint 2
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint 3
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fixup
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Apply suggestions from code review
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Unfortunately go doesn't always ensure that execd processes are completely
waited for. On linux this means that zombie processes can occur.
This PR ensures that these are waited for by using signal notifier in serv and
passing a context elsewhere.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Upgrade Gliderlabs SSH to 0.3.3 and add FailedConnectionCallback
Following the merging of https://github.com/gliderlabs/ssh/pull/143 we
can now report connections to the ssh server that have failed before
public key exchange has completed using the standard fail2ban message.
This PR updates Gliderlabs SSH and adds a callback that will provide this
logging.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* move the callback to its own function to make the logging appear little nicer
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Set self-adjusting deadline for connection writing
In #16055 it appears that the simple 5s deadline doesn't work for large
file writes. Now we can't - or at least shouldn't just set no deadline
as go will happily let these connections block indefinitely. However,
what seems reasonable is to set some minimum rate we expect for writing.
This PR suggests the following algorithm:
* Every write has a minimum timeout of 5s (adjustable at compile time.)
* If there has been a previous write - then consider its previous
deadline, add half of the minimum timeout + 2s per kb about to written.
* If that new deadline is after the minimum timeout use that.
Fix#16055
* Linearly increase timeout
* Make PerWriteTimeout, PerWritePerKbTimeouts configurable
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* Make SSH server host key path configurable
* make it possible to have multiple keys
* Make gitea.rsa the default key
* Add some more logging
Signed-off-by: Andrew Thornton <art27@cantab.net>
* remove github.com/unknwon/com from models
* dont use "com.ToStr()"
* replace "com.ToStr" with "fmt.Sprint" where its easy to do
* more refactor
* fix test
* just "proxy" Copy func for now
* as per @lunny
Continuing on from #13953 continue to improve and standardise
logging from internal SSH.
Also updates the fail2ban setup
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR standardizes reporting of permission denied from the internal ssh.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* Log IP on SSH authentication failure
fixes https://github.com/go-gitea/gitea/issues/13094
* include string 'Failed authentication attempt' in error
* update fail2ban docs
also match failed authentication over command line
* better logging of authentication errors with IP addresses
* format ...
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
* Add ssh certificate support
* Add ssh certificate support to builtin ssh
* Write trusted-user-ca-keys.pem based on configuration
* Update app.example.ini
* Update templates/user/settings/keys_principal.tmpl
Co-authored-by: silverwind <me@silverwind.io>
* Remove unused locale string
* Update options/locale/locale_en-US.ini
Co-authored-by: silverwind <me@silverwind.io>
* Update options/locale/locale_en-US.ini
Co-authored-by: silverwind <me@silverwind.io>
* Update models/ssh_key.go
Co-authored-by: silverwind <me@silverwind.io>
* Add missing creation of SSH.Rootpath
* Update cheatsheet, example and locale strings
* Update models/ssh_key.go
Co-authored-by: zeripath <art27@cantab.net>
* Update models/ssh_key.go
Co-authored-by: zeripath <art27@cantab.net>
* Update models/ssh_key.go
Co-authored-by: zeripath <art27@cantab.net>
* Update models/ssh_key.go
Co-authored-by: zeripath <art27@cantab.net>
* Update models/ssh_key.go
* Optimizations based on feedback
* Validate CA keys for external sshd
* Add filename option and change default filename
Add a SSH_TRUSTED_USER_CA_KEYS_FILENAME option which default is
RUN_USER/.ssh/gitea-trusted-user-ca-keys.pem
Do not write a file when SSH_TRUSTED_USER_CA_KEYS is empty.
Add some more documentation.
* Remove unneeded principalkey functions
* Add blank line
* Apply suggestions from code review
Co-authored-by: zeripath <art27@cantab.net>
* Add SSH_AUTHORIZED_PRINCIPALS_ALLOW option
This adds a SSH_AUTHORIZED_PRINCIPALS_ALLOW which is default
email,username this means that users only can add the principals
that match their email or username.
To allow anything the admin need to set the option anything.
This allows for a safe default in gitea which protects against malicious
users using other user's prinicipals. (before that user could set it).
This commit also has some small other fixes from the last code review.
* Rewrite principal keys file on user deletion
* Use correct rewrite method
* Set correct AuthorizedPrincipalsBackup default setting
* Rewrite principalsfile when adding principals
* Add update authorized_principals option to admin dashboard
* Handle non-primary emails
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add the command actually to the dashboard template
* Update models/ssh_key.go
Co-authored-by: silverwind <me@silverwind.io>
* By default do not show principal options unless there are CA keys set or they are explicitly set
Signed-off-by: Andrew Thornton <art27@cantab.net>
* allow settings when enabled
* Fix typos in TrustedUserCAKeys path
* Allow every CASignatureAlgorithms algorithm
As this depends on the content of TrustedUserCAKeys we should allow all
signature algorithms as admins can choose the specific algorithm on their
signing CA
* Update models/ssh_key.go
Co-authored-by: Lauris BH <lauris@nix.lv>
* Fix linting issue
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Change graceful to use a singleton obtained through GetManager instead of a global.
* Graceful: Make TestPullRequests shutdownable
* Graceful: Make the cron tasks graceful
* Graceful: AddTestPullRequest run in graceful ctx
* Graceful: SyncMirrors shutdown
* Graceful: SetDefaultContext for Xorm to be HammerContext
* Avoid starting graceful for migrate commands and checkout
* Graceful: DeliverHooks now can be shutdown
* Fix multiple syncing errors in modules/sync/UniqueQueue & Make UniqueQueue closable
* Begin the process of making the repo indexer shutdown gracefully
* Graceful Shutdown for windows and others
Restructures modules/graceful, adding shutdown for windows, removing and
replacing the old minwinsvc code.
Creates a new waitGroup - terminate which allows for goroutines to
finish up after the shutdown of the servers.
Shutdown and terminate hooks are added for goroutines.
* Remove unused functions - these can be added in a different PR
* Add startup timeout functionality
* Document STARTUP_TIMEOUT
* Prevent deadlock in indexer initialisation during graceful restart
* Move from gracehttp to our own service to add graceful ssh
* Add timeout for start of indexers and make hammer time configurable
* Fix issue with re-initialization in indexer during tests
* move the code to detect use of closed to graceful
* Handle logs gracefully - add a pid suffix just before restart
* Move to using a cond and a holder for indexers
* use time.Since
* Add some comments and attribution
* update modules.txt
* Use zero to disable timeout
* Move RestartProcess to its own file
* Add cleanup routine
resolves git conflicts from #3896 (credit to @belak, in case github doesn't keep original author during squash)
Co-Authored-By: Matti Ranta <techknowlogick@gitea.io>
* Panic don't fatal on create new logger
Fixes#5854
Signed-off-by: Andrew Thornton <art27@cantab.net>
* partial broken
* Update the logging infrastrcture
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Reset the skip levels for Fatal and Error
Signed-off-by: Andrew Thornton <art27@cantab.net>
* broken ncsa
* More log.Error fixes
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove nal
* set log-levels to lowercase
* Make console_test test all levels
* switch to lowercased levels
* OK now working
* Fix vetting issues
* Fix lint
* Fix tests
* change default logging to match current gitea
* Improve log testing
Signed-off-by: Andrew Thornton <art27@cantab.net>
* reset error skip levels to 0
* Update documentation and access logger configuration
* Redirect the router log back to gitea if redirect macaron log but also allow setting the log level - i.e. TRACE
* Fix broken level caching
* Refactor the router log
* Add Router logger
* Add colorizing options
* Adjust router colors
* Only create logger if they will be used
* update app.ini.sample
* rename Attribute ColorAttribute
* Change from white to green for function
* Set fatal/error levels
* Restore initial trace logger
* Fix Trace arguments in modules/auth/auth.go
* Properly handle XORMLogger
* Improve admin/config page
* fix fmt
* Add auto-compression of old logs
* Update error log levels
* Remove the unnecessary skip argument from Error, Fatal and Critical
* Add stacktrace support
* Fix tests
* Remove x/sync from vendors?
* Add stderr option to console logger
* Use filepath.ToSlash to protect against Windows in tests
* Remove prefixed underscores from names in colors.go
* Remove not implemented database logger
This was removed from Gogs on 4 Mar 2016 but left in the configuration
since then.
* Ensure that log paths are relative to ROOT_PATH
* use path.Join
* rename jsonConfig to logConfig
* Rename "config" to "jsonConfig" to make it clearer
* Requested changes
* Requested changes: XormLogger
* Try to color the windows terminal
If successful default to colorizing the console logs
* fixup
* Colorize initially too
* update vendor
* Colorize logs on default and remove if this is not a colorizing logger
* Fix documentation
* fix test
* Use go-isatty to detect if on windows we are on msys or cygwin
* Fix spelling mistake
* Add missing vendors
* More changes
* Rationalise the ANSI writer protection
* Adjust colors on advice from @0x5c
* Make Flags a comma separated list
* Move to use the windows constant for ENABLE_VIRTUAL_TERMINAL_PROCESSING
* Ensure matching is done on the non-colored message - to simpify EXPRESSION
* use native golang SSH library but ssh-keygen when enable built-in SSH server to remove dependent on that command lines
* fix tests and add comment head
Fix#5765 by log the IP address of a connecting remote machine in case of a SSH connection error for the built-in ssh server.
Signed-off-by: Robert Sprunk <github@sprunk.me>
* Configurable SSH cipher suite
* Update configuration file comment
* Add default in settings loading code
* Fix fmt and log messsage
* Remove default from code as this could probably might not be good idea
This change corrects a few logging issues:
* Standardized formatting errors with '%v'.
* Standardized failure warning word usage.
* Corrected an instance of using the standard log library when
the gitea log library should be used instead.
When use builtin SSH server with concurrent operations, there are probabilities
One connection could use the command from another connections.
Fix this by set SSH_ORIGINAL_COMMAND for each command, not set in global scope.
According to the docs [1], the Reply method must be called for all
requests where WantReply is true. This fixes a hanging java ssh
implementation (jsch) which sets WantReply flag and waits for reply from
the server.
[1] https://godoc.org/golang.org/x/crypto/ssh#Request.Reply
