Xenstore

XenStore is a database, hosted by domain 0, that supports transactions and atomic operations. XenStore is actually located in a single file, /var/lib/xenstored/tdb, on Domain0. (tdb stands for Tree Database.)

 

Dom 0的用户态进程xenstored起着分发汇总各个domain（包括Dom 0自身）消息的核心枢纽作用，这些消息包含各种操作（读写tdb文件，增加watch等）。每个消息都是一类。

struct xsd_sockmsg

{

    uint32_t type;  /* XS_??? */

    uint32_t req_id;/* Request identifier, echoed in daemon's response.  */

    uint32_t tx_id; /* Transaction id (0 if not related to a transaction). */

    uint32_t len;   /* Length of data following this. */

 

    /* Generally followed by nul-terminated string(s). */

};

为了增加message传递的效率，每个guest和domain 0之间都一个共享的页面，核心数据结构是

struct xenstore_domain_interface {

    char req[XENSTORE_RING_SIZE]; /* Requests to xenstore daemon. */

    char rsp[XENSTORE_RING_SIZE]; /* Replies and async watch events. */

    XENSTORE_RING_IDX req_cons, req_prod;

    XENSTORE_RING_IDX rsp_cons, rsp_prod;

};

见下面的xen_store_evtchn和xen_store_mfn小节。

 

参考资料：

docs/misc/xenstore.txt 源代码树中xen 4.1版本

值得一看

 

的流程

XenStore：使用，结构和原理(3. XenStore启动过程分析)  独立的分析文章，不错

The Xenstored daemon maintains a registry of information including memory and event channel links between Domain 0 and all other Domain U Guests. The Domain 0 virtual machine leverages this registry to setup device channels with other virtual machines on the system. (See Domain 0 to Domain U Communication for more details).

 

实现见tools/xenstore,最核心文件是xenstored_core.c

get_node read_node==>tdb_fetch

 

 

来自

The Xenstored service is a single-threaded daemon, running in the control domain. It is the central point of communication, as guests communicate to each others using it. Basically, it is a file-systemlike database, where control data is hierarchically organized.

There is two different ways a client can connect and communicate with the Xenstored service:

• First, using the ring buffers. These are two circular buffers stored inside in the guests’ memory: one for sending request to the Xenstored service and one for reading its replies.

• Second, using the Unix sockets. These are accessible only for processes running inside the control domain. Clients of the Xenstored service use the standard Unix read and write operations to access to the database.

信息的访问方式

Xenstore is accessible by either a Unix domain socket in Domain-0, a kernel-level API, or an ioctl interface via /proc/xen/xenbus.下面详细讨论

和xen_store_mfn

Xen 3.1

首先每个domain（包括Dom 0）都有自己独立的一份xen_store_evtchn和xen_store_mfn，而每个domain的xen_store_mfn都和Dom 0共享。由于Dom 0的特殊性，所以Dom 0的xen_store_evtchn和xen_store_mfn创建方式和Dom U不同，见xenbus_probe_init。Dom U必须由Dom 0创建，所以Dom 0会预先为Dom U分配一个xen_store_evtchn（注意是在Dom U中分配evtchn，而不是在Dom 0中）和xen_store_mfn（注意是在Dom U中分配mfn而不是在Dom 0中），然后再通过start_info传递给Dom U。

 

Interdomain之间通信必须两个Dom各自创建创建一个evtchn，然后bind。各guest domain和Dom 0的xenstored通信，首先各guest domain的evtchn都是xen_store_evtchn(注意各个Domain的xen_store_evtchn值不一定相同，而且都属于各自的domain）。

然后在Dom 0中，为各个Dom U分配一个evtchn对应guest的xen_store_evtchn，实现interdomain之间的通信。代码见new_domain=> xc_evtchn_bind_interdomain(xce_handle, domid, port);

下面解释一下xc_evtchn_bind_interdomain，

在int domain_init(void)函数中xce_handle就是文件/dev/xen/evtchn的handle。有语句如下：

xce_handle = xc_evtchn_open();

/dev/xen/evtchn allows a usermode program（如xenstored） to wait for specific event channels.

/dev/xen/evtchn 实现见linux-2.6-xen-sparse/drivers/xen/evtchn/evtchn.c。

xc_evtchn_bind_interdomain=>ioctl(xce_handle, IOCTL_EVTCHN_BIND_INTERDOMAIN, &bind)=> evtchn_ioctl, evtchn_ioctl的代码如下：

case IOCTL_EVTCHN_BIND_INTERDOMAIN: {

              struct ioctl_evtchn_bind_interdomain bind;

              struct evtchn_bind_interdomain bind_interdomain;

 

              rc = -EFAULT;

              if (copy_from_user(&bind, uarg, sizeof(bind)))

                     break;

 

              bind_interdomain.remote_dom  = bind.remote_domain;

              bind_interdomain.remote_port = bind.remote_port;

              rc = HYPERVISOR_event_channel_op(EVTCHNOP_bind_interdomain,

                                           &bind_interdomain);

              if (rc != 0)

                     break;

 

              rc = bind_interdomain.local_port;

              evtchn_bind_to_user(u, rc);

              break;

       }

特别值得注意的是evtchn_bind_to_user，这里是复用，多个guest都通过设备/dev/xen/evtchn与xenstored交换信息，后面提到的evtchn_device_upcall就是解复用过程（demux）。

 

在Xenstored中如何访问Dom U的xen_store_mfn？

Dom0中执行的do_introduce有如下语句：

interface = xc_map_foreign_range(

                     *xc_handle, domid,

                     getpagesize(), PROT_READ|PROT_WRITE, mfn);

实现了把Dom U的mfn映射到Dom 0的空间。

xc_handle是通过xc_interface_open打开的/proc/xen/privcmd。

 

在Xenstored中如何访问Dom 0的xen_store_mfn？

见dom0_init函数，通过下面两个特殊文件把Dom 0的xen_store_evtchn和xen_store_mfn传递给xenstored。

#define XENSTORED_PROC_KVA  "/proc/xen/xsd_kva"

#define XENSTORED_PROC_PORT "/proc/xen/xsd_port"

访问方式（1）－核心态通信xenbus

下面从两个方向讨论通信

(1)从Dom U kernel到Dom 0 xenstored

Dom U：

上层 xenbus API=>xb_write/xb_read=>notify_remote_via_evtchn(xen_store_evtchn);

Dom 0：

evtchn_do_upcall=> evtchn_device_upcall(port);=>xenstored(通过select被触发，见xenstored_core.c 中的main函数)

 

下面解释一下evtchn_device_upcall，

(2) 从Dom 0 xenstored到Dom U kernel

Dom 0：

xenstored_core.c中的main函数=>handle_input/handle_output=>readchn/writechn(见new_domain函数中的初始化)=> xc_evtchn_notify(xce_handle, conn->domain->port);

 

Dom U:

xb_init_comms中有如下语句：

err = bind_caller_port_to_irqhandler(

              xen_store_evtchn, wake_waiting,

              0, "xenbus", &xb_waitq);

这样wake_waiting将唤醒下面的xenbus_thread线程或者在xb_write阻塞的进程

 

xs_init函数中有如下语句：

ask = kthread_run(xenwatch_thread, NULL, "xenwatch");

task = kthread_run(xenbus_thread, NULL, "xenbus");

xenbus_thread=> process_msg  而process_msg会唤醒xenwatch_thread

访问方式（2）Dom 0的用户态访问

1.         xs_daemon_open(): connect to socket /var/run/xenstored/socket or /var/run/xenstored/socket_ro (for readonly operations) for read/write.

2.         Xend in dom0 will connect to xenstored via xs_daemon_open(): it avoids a context switch between user space and kernel space, thus more efficient.

访问方式（3）Dom U的用户态访问

下面来自[Xen-devel] Why xs_domain_open() in fs_backend的邮件讨论

xs_domain_open() is required for xenstore clients to work when they're not in the same domain as xenstored (either because xenstore is in a stub domain or because the client is not in dom0).

 

The xenstore clients work fine in any domain which exports a /proc/xen/xenbus interface to userspace, subject to the user running them having sufficient guest privilege to access the magic device and subject to the permissions within xenstore allowing the domain to list the directories in question etc.

 

Dom U 通过文件/proc/xen/xenbus或/dev/xen/xenbus访问xenstore，见函数

const char *xs_domain_dev(void)

{

       char *s = getenv("XENSTORED_PATH");

       if (s)

              return s;

 

#ifdef __linux__

       return "/proc/xen/xenbus";

#else

       return "/dev/xen/xenbus";

#endif

}

/proc/xen/xenbus 设备文件的实现见linux-2.6-xen-sparse/drivers/xen/xenbus/xenbus_dev.c

 * Driver giving user-space access to the kernel's xenbus connection

 * to xenstore.

 

xenbus的用户（如driver）通过register_xenbus_watch注册watch，该类XS_WATCH信息发送到xenstored登记，watch的全盘实现见tools/xenstore/ xenstored_watch.c