[PATCH STABLE] worker: wait worker pid explicitly

[Jun Wu]

Excerpts from Yuya Nishihara's message of 2016-07-25 21:49:35 +0900:
> On Mon, 25 Jul 2016 10:27:17 +0100, Jun Wu wrote:
> > Excerpts from Yuya Nishihara's message of 2016-07-24 15:32:10 +0900:
> > > Also it doesn't work under vanilla commandserver which replaces ui.fout by
> > > a pseudo file object.
> > 
> > I think commandserver needs some lock mechanism to protect its I/O. flock
> > is a good choice wherever supported. The problem is not only limited to
> > workers but also any other places using threads / processes.
> 
> No idea how flock() will work for duplicated fds. Are we going to make a
> temporary lock file?

TIL flock and fcntl(F_SETLKW) are not the same. The latter works in this
case without an extra file.
 
> As long as underlying fwrite() is thread-safe, the issue can be avoided
> by writing header + data by single write() call. That can't be true for
> fork(), but IIRC worker.py is the only place we have to care.

Ideally, workers are threads. But I understand we use processes to
workaround Python's GIL. It feels like reinventing stdio because of the GIL.

The main thing I dislike about the pipe plan is unnecessary memory copy /
CPU usage. If the files being updated are many, their paths can be a lot of
bytes.

Assuming status lines are less than 4KB, make sure write uses a string
ending with "\n" (aka. line buffered, stdbuf -oL) should solve this issue
without any locks.