some notes when studying workqueue

1. for every workqueue_struct, there are CPU_NUMBER cpu_workqueue_struct. For unbound case, there is only one cpu_workqueue_struct.

2. global_cwq->worklist is used queue the pending works, it is to say that queue_work() will insert the work_struct into this worklist finally.

3. work_on_cpu() in workqueue.c maybe a good example/reference for programer.

4. every worker comprehends a worker_thread which runs the work functions really, and the thread is bind to the corresponding cpu by kthread_bind().

5.print_symbol() is a good tool for debuging, which can be used to print the function name.

6.maybe_destroy_workers(), if there are too many idle workers(threads), and the thread stays in idle longer than 300*HZ(5 minutes), it will destroy this idle worker.

7.worker->schedule, this list used to queue the WORK_STRUCT_LINKED work_struct, and I don't see any function/purpose of this mechnism. Now I know how it works, reference following Item3, it is used to avoid the same work run simultaneously on the same CPU, if one work has been running, and another same work has been insert the queue, and other thread may take down it and run it, certainly, it should be forbiddened, so it will not run and just queue it on worker-> schedule, the worker is the one which is running it previously( A single work shouldn't be executed concurrently by multiple workers on a single cpu.)  

8.for every worker_thread() which is used to run the work_struct->func(), when enter/exit this function, both will call manage_workers(), and it will call maybe_destroy_workers() and maybe_create_worker().

 maybe_destroy_workers: will kill the redundant worker(threads).

 maybe_create_worker: will create the worker(threads) if necessary. If the gcwq->worklist is unempty and more than one worker(thread) is running, plus there is no idle worker, when above conditions are satified, it will create a new worker(thread).

9.WORK_STRUCT_PENDING_BIT will guarantee that there is no more than one work is queued at any time.

10. for WQ_HIGHPRI, which is used in workqueue, for this type workqueue, work will be inserted into the front position in gcwq->worklist(work pending list), and when the corresponding work get opportunity to run, it also will check whether next work is a WQ_HIGHPRI one, if so, driver will wake up another thread(worker) to run it.(reference process_one_work()).

11.gcwq_nr_running is used to indicate how many workers(threads) in the corresponding gcwq(CPU)are running.

12. nr_active is used to indeicate how many works of cpu_workqueue_struct have been inserted into the queues, including on gcwq->worklist and worker->schedule, not include cwq->delayed_works. it will be decreased only after the func has been finished.

quesiton:

1. for one cpu, there maybe more than one worker(thread) run at the same time, they all take down the work from the gcwq->worklist, how to make sure the serialization?

2. ordered is only make sure all the works on the same workqueue_struct run orderedly.

3. WQ_NON_REENTRANT just only make sure that the same work is queued on the same gcwq->worklist, and when dealing with the work in process_one_work(), it will also check whether other workers in the same CPU is running the work(using __find_worker_executing_work()) to avoid more threads run the same work simultaneously, it is also the answer for the question 1.

summary: to make sure the same work only runs one at any time, use NONE_REENTRANT is enough.

         to make sure ever work on the same work_queue runs one at any time, use ORDERED is enough, certainly, it will cost more than NONE_REENTRANT. ORDERED is more like the old interface: create_singlethread_workqueue().

for ORDERED work, if one work has been queued, the new work will be insert into cwq->delayed_works, and it will be ddealed with by process_one_work()->cwq_dec_nr_in_flight()->cwq_activate_first_delayed().