summaryrefslogtreecommitdiff
path: root/tools/merge_fetch.py
Commit message (Collapse)AuthorAgeFilesLines
* Don't crash in merge_fetch when there's a logorder file.Linus Nordberg2016-12-021-5/+7
| | | | | The 'logorder' file keeps hashes hexencoded. When read into the logorder list they're decoded.
* Don't get new entries while we still have entries to fetch.Linus Nordberg2016-12-021-3/+4
|
* Drain pipe between fetch_send control process and worker processes.Linus Nordberg2016-12-011-32/+55
| | | | | The pipe is limited so send blocks. Drain after _every_ send in _both_ directions.
* Remove spurious message length checks.Linus Nordberg2016-12-011-9/+1
| | | | No only are they spurious but even harmful, by hiding errors.
* Don't crash when restarting merge_fetch worker process.Linus Nordberg2016-12-011-5/+7
| | | | Store storagenode object in procs as well.
* Don't pass 0 to multiprocessing.Pipe.poll().Linus Nordberg2016-11-281-2/+2
| | | | Documentation says to pass nothing for no blocking.
* Dedup log initialisation code.Linus Nordberg2016-11-281-7/+2
|
* Fix a crash bug; improve IPC; use cycle().Linus Nordberg2016-11-271-40/+40
| | | | | | | | | Workers: - Consume all messages each round, just as control process does. Control process: - Use itertools.cycle for the circular list. - Fix bug where an entry could be fetched multiple times, crashing when updating fetch_dict.
* Improve two comments.Linus Nordberg2016-11-261-2/+6
|
* Add back chunking for get_entries in parallelised merge_fetch.Linus Nordberg2016-11-261-10/+9
| | | | | Even if it's not important for parallelism on the merge node, it's not a bad idea to keep the requests from growing.
* Remove old assertion used for testing.Linus Nordberg2016-11-261-2/+0
|
* Bug fixes from map.Linus Nordberg2016-11-261-5/+4
|
* merge_fetch: Write to chainsdb in controlling process.Linus Nordberg2016-11-251-73/+74
| | | | | | | Multiple writers won't work when using permdb. Also, improve throughput by polling worker processes with timeout 0 and read all messages at once.
* Manual rebase onto master with map/robust-distribution.Linus Nordberg2016-11-251-28/+219
|
* Add a bit of documentation.Linus Nordberg2016-11-251-1/+4
|
* Use sessions in merge_fetchMagnus Ahltorp2016-11-181-14/+16
|
* Added permdb support to mergeMagnus Ahltorp2016-07-111-4/+5
|
* Remove debug printouts from merge programs.Linus Nordberg2015-11-201-1/+1
|
* Fix _args -> args since we now use it.Linus Nordberg2015-11-101-1/+1
|
* Print the string "timing" and the name of the program in merge timing info.Linus Nordberg2015-11-101-1/+1
|
* Print timing information in merge_fetch too.Linus Nordberg2015-11-101-0/+4
|
* Merge is now run by shell script tools/merge.Linus Nordberg2015-11-101-12/+36
| | | | | | | tools/merge run merge_fetch.py, merge_backup.py, merge_sth.py and merge_dist.py sequentially. TODO: test backupquorum != 0
* Split merge.py into three pieces.Linus Nordberg2015-11-101-0/+97