summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* initlog.py: Consult 'logorder'; Create chainsdb.pmerge2-rebasedLinus Nordberg2016-12-051-3/+25
|
* merge_dist: Prefix logging lines with nodename.Linus Nordberg2016-12-041-12/+15
|
* merge_backup, merge_dist: Bring child processes down on TERM.Linus Nordberg2016-12-032-2/+15
|
* merge_backup: Print timing info for each secondary.Linus Nordberg2016-12-031-17/+21
| | | | | Also, prefix all log entries with the name of the secondary merge node operating on.
* Open permdb read-only in merge_backup and merge_dist.Linus Nordberg2016-12-033-6/+8
|
* merge_dist: Distribute independently to each frontend node.Linus Nordberg2016-12-031-48/+77
|
* Minor certtools cleanup.Linus Nordberg2016-12-031-7/+2
|
* merge_sth: Don't try to go back in time.Linus Nordberg2016-12-033-2/+7
| | | | | | | | | | | Due to a recent change in get_nfetched(), merge_sth lacking the 'sth' file could go back to a tree size of zero. It's rejected by frontend nodes with a newer tree, but it's still a severe error to produce that piece of data. We add tools/initlog.py which creates an 'sth' file with tree size 0 and have tests invoke it after starting the erlang nodes, before starting the continous merge processes.
* Add initlog.py.Linus Nordberg2016-12-023-13/+102
|
* Don't crash in merge_fetch when there's a logorder file.Linus Nordberg2016-12-021-5/+7
| | | | | The 'logorder' file keeps hashes hexencoded. When read into the logorder list they're decoded.
* Don't get new entries while we still have entries to fetch.Linus Nordberg2016-12-021-3/+4
|
* Drain pipe between fetch_send control process and worker processes.Linus Nordberg2016-12-011-32/+55
| | | | | The pipe is limited so send blocks. Drain after _every_ send in _both_ directions.
* Remove spurious message length checks.Linus Nordberg2016-12-011-9/+1
| | | | No only are they spurious but even harmful, by hiding errors.
* Don't crash when restarting merge_fetch worker process.Linus Nordberg2016-12-011-5/+7
| | | | Store storagenode object in procs as well.
* Documentation changes.Linus Nordberg2016-12-011-3/+4
|
* Separate two error cases when reading 'verified' file.Linus Nordberg2016-12-011-1/+6
|
* Increase timeouts waiting for merge.Linus Nordberg2016-12-011-3/+3
| | | | | | | Increase from 15 to 40 seconds. In theory, it should never be more than 20 seconds plus actual time to do the job with a merge interval of 10 seconds. Max wait times are 3+1 seconds in fetch, 3s in backup, 10s in sth and 3s in dist.
* Increase merge timeouts.Linus Nordberg2016-11-301-3/+3
|
* Parallelise merge_dist.Linus Nordberg2016-11-303-60/+97
| | | | Also deduplicate some code.
* Clean up a bit.Linus Nordberg2016-11-281-17/+3
|
* If any backup process failed, try again.Linus Nordberg2016-11-281-3/+7
| | | | | | Try again without waiting for an update of the fetched file, which won't happen in the tests (but probably will happen in real life, masking this bug).
* Do remove process from procs dict when they fail too.Linus Nordberg2016-11-281-4/+4
|
* Don't pass 0 to multiprocessing.Pipe.poll().Linus Nordberg2016-11-281-2/+2
| | | | Documentation says to pass nothing for no blocking.
* Dedup log initialisation code.Linus Nordberg2016-11-285-27/+16
|
* Parallelise merge_backup.Linus Nordberg2016-11-271-45/+94
| | | | | We're still failing the tests when the merge secondary goes away, sometimes.
* Fix a crash bug; improve IPC; use cycle().Linus Nordberg2016-11-271-40/+40
| | | | | | | | | Workers: - Consume all messages each round, just as control process does. Control process: - Use itertools.cycle for the circular list. - Fix bug where an entry could be fetched multiple times, crashing when updating fetch_dict.
* Improve two comments.Linus Nordberg2016-11-261-2/+6
|
* Add back chunking for get_entries in parallelised merge_fetch.Linus Nordberg2016-11-261-10/+9
| | | | | Even if it's not important for parallelism on the merge node, it's not a bad idea to keep the requests from growing.
* Remove old assertion used for testing.Linus Nordberg2016-11-261-2/+0
|
* Log simple web request logging on level infoMagnus Ahltorp2016-11-261-2/+6
| | | | This change was lost in 616be2a.
* Added back support for R16Magnus Ahltorp2016-11-261-0/+10
| | | | This change was lost in 616be2a.
* Don't need this bc don't do locking anymore.Linus Nordberg2016-11-261-1/+0
| | | | Also, it collides with permdb.
* Bug fixes from map.Linus Nordberg2016-11-261-5/+4
|
* merge_fetch: Write to chainsdb in controlling process.Linus Nordberg2016-11-252-104/+74
| | | | | | | Multiple writers won't work when using permdb. Also, improve throughput by polling worker processes with timeout 0 and read all messages at once.
* Sleep less in tests.Linus Nordberg2016-11-253-28/+39
|
* Do import datetime even more.Linus Nordberg2016-11-253-5/+8
|
* Manual rebase onto master with map/robust-distribution.Linus Nordberg2016-11-2519-210/+718
|
* Do import datetime.Linus Nordberg2016-11-251-1/+2
| | | | How did this ever work?
* Add a bit of documentation.Linus Nordberg2016-11-255-7/+19
|
* Merge remote-tracking branch 'refs/remotes/map/robust-distribution'Linus Nordberg2016-11-2513-199/+379
|\
| * Add session capability to unauth web requestsMagnus Ahltorp2016-11-221-6/+9
| |
| * Change hard coded paths to use variablesMagnus Ahltorp2016-11-221-8/+8
| |
| * Log simple web request logging on level infoMagnus Ahltorp2016-11-221-2/+6
| |
| * Added back support for R16Magnus Ahltorp2016-11-221-0/+10
| |
| * parsebench: change from table to div:sMagnus Ahltorp2016-11-221-4/+4
| |
| * Only allocate one color per stageMagnus Ahltorp2016-11-191-13/+36
| |
| * Use sessions in merge_fetchMagnus Ahltorp2016-11-182-16/+18
| |
| * Change algorithm for merge backup and merge distMagnus Ahltorp2016-11-024-149/+185
| |
| * Added performance testsMagnus Ahltorp2016-11-024-8/+90
| |
| * Send list of entries to frontendMagnus Ahltorp2016-11-023-30/+50
| | | | | | | | Handle connection error
generated by cgit v1.1 at 2024-12-25 05:10:56 +0000