(PDF) Minimizing Computing-plus-Communication Energy Consumptions in Virtualized Networked Data Centers

2016 IEEE Symposium on Computers and Communication (ISCC) Minimizing Computing-plus-Communication Energy Consumptions in Virtualized Networked Data Centers Mohammad Shojafar∗ , Claudia Canali∗ , Riccardo Lancellotti∗ and Enzo Baccarelli† ∗ Departmentof Engineering ”Enzo Ferrari”, University of Modena and Reggio Emilia, Email:{mohammad.shojafar, claudia.canali, riccardo.lancellotti}@unimore.it † Department of Information Engineering and Telecommunication, Sapienza University of Rome Email: {enzo.baccarelli}@uniroma1.it Abstract—In this paper, we propose a dynamic resource However, current proposals are still far from solving all provisioning scheduler to maximize the application through- the problems that characterize the management of large cloud put and minimize the computing-plus-communication energy data centers. For example, several existing solutions [2]–[4] consumption in virtualized networked data centers. The goal is to maximize the energy-efficiency, while meeting hard QoS rely on the forecast of future workload and infrastructure requirements on processing delay. The resulting optimal resource resource demands. This may be a problem due to inherent scheduler is adaptive, and jointly performs: i) admission control errors in the prediction steps, that may be exacerbated by of the input traffic offered by the cloud provider; ii) adap- the presence of highly variable and unpredictable fluctuations tive balanced control and dispatching of the admitted traffic; of the workload patterns. Even solutions that do not rely on iii) dynamic reconfiguration and consolidation of the Dynamic Voltage and Frequency Scaling (DVFS)-enabled virtual machines prediction have the critical shortcoming of considering only instantiated onto the virtualized data center. The proposed the resources of the servers, without considering the power scheduler can manage changes of the workload without requiring consumption of the underlying intra-data center network [5]. A server estimation and prediction of its future trend. Furthermore, specular set of works focuses on the network without providing it takes into account the most advanced mechanisms for power a detailed view of the computational infrastructure [6]–[8]. reduction in servers, such as DVFS and reduced power states. Performance of the proposed scheduler is numerically tested and Only recently, the authors proposed some solutions to manage compared against the corresponding ones of some state-of-the- data centers considering both computational and networking art schedulers, under both synthetically generated and measured infrastructure [9]. real-world workload traces. The results confirm the delay-vs.- In this paper, we present a further step with respect to energy good performance of the proposed scheduler. these proposals by introducing an innovative mechanism that Index Terms—Virtualized Data Centers, Cloud Computing, combines admission control, request management through Resource Allocation, Energy-efficiency, Lyapunov Optimization. scheduling, and server management by using DVFS and re- duced power states of the servers. The final goal is to manage a I. I NTRODUCTION cloud networked data center in an energy-efficient way, while meeting QoS requirements. The main contributions of our Over the last years, the evolution of demands for mod- proposal are the following ones: (1) we manage unpredictable ern applications combined with the adoption of the cloud changes of workload without the need for estimation and computing paradigm has led to the establishment of large- prediction mechanisms; (2) we consider virtual machine (VM) scale virtualized data centers. They are characterized by a consolidation and task replacements; (3) we take into account large (and ever increasing) number of servers, which rely both the computing (including power state changes and DVFS) on hardware-assisted virtualization and multi-hop networks to and the communication energies in the management of the provide server interconnection [1]. infrastructure. In a Software-as-a-Service (SaaS) scenario, two main needs Our proposal is tested using a simulative approach based must be taken into account. First, QoS requirements, typically on both real traces and synthetic workloads. The scheduler expressed in the form of a Service Level Agreements (SLAs), is tested under different data center scenarios. Furthermore, must be met. Second, we also aim to reduce the energy its performance is compared with the ones of the recent consumption, in order to meet the demand of the so-called GRADient-based Iterative Scheduler (GRADIS) in [10], the green computing (or, more pragmatically, we need to reduce Static Lyapunov-based Scheduler (SLS) in [11], the hybrid power consumption to maximize the economic revenue). The NetDC scheduler (H-NetDC) in [12], and the NetDC scheduler perspective of this paper is that of an entity that is in in [9] by considering throughput, delay, and energy costs charge of providing services to clients and controls the cloud (e.g., computing plus communication costs). Our experiments infrastructure as well. This scenario is typical when a SaaS prove that our proposal maximizes energy efficiency, while provider manages its own physical infrastructure, or when a per-application service rate is also maximized according to company deploys its own private cloud platform. the control policies we define for each server. 978-1-5090-0679-3/16/$31.00 ©2016 IEEE 2016 IEEE Symposium on Computers and Communication (ISCC) Clients Networked data center Networked Data Center Server #1 VM11 Admission Traffic control R11(t) Admission Servers control & dispatching μ11(t) controller reconfiguration A1(t) ADC R1(t) U11(t) & consolidation #1 RN1(t) VMN1 Clients W1(t) μN1(t) VMM #1 Data center UN1(t) Traffic A2(t) network R2(t) Processing ADC controller & #2 dispatcher Servers W2(t) Server #M VM1M R1M(t) ... AN(t) Data center ADC RN(t) μ1M(t) #N U1M(t) network WN(t) VMNM RNM(t) μNM(t) VMM #M Input Admitted Dispatched UNM(t) Fig. 1: Data center overview traffic traffic traffic The rest of this paper is organized as follows. After pre- Fig. 2: Data center detailed model senting the model of the considered virtualized data center architecture in Section II, we provide the problem formulation the twofold task of storing the allowed requests in buffers and and the proposed joint scheduler in Section III. Performance distributing them over the VMs that will be responsible for evaluation results obtained through extensive simulation tests their processing. Each VM has a queue that is used to store are presented in Section IV. Finally, Section V presents some requests to be processed. Furthermore, a per-server VMM is concluding remarks. responsible for both managing the status of the server (power ON/OFF, CPU frequency that is, servers reconfiguration) and II. S YSTEM A RCHITECTURE for deciding which VMs must run on each server (that is, A. Networked data center overview servers consolidation). We begin to present an overview of the considered net- We consider that the data center supports a Software as worked data center that uses our solution for performing a Service (SaaS) cloud system, that provides a set A of scheduling and admission control. applications. Let N = |A| be the number of such applications. Fig. 1 provides a general scheme of the considered net- The VMs that support the execution of these applications worked data center. We have an heterogeneous set of clients are deployed over a set S of M servers. For this purpose, issuing requests to the networked data center (e.g., the boxes we assume that each server j ∈ S has a set of resources on the left side of the figure). The set of requests is first (i.e., CPU, disk, memory, etc.) that are binded to the VMs processed by the admission controller that decides whether hosted on it and supervised by its VMM. In practice, this or not the request should be accepted by the data center. VMM resides on the host Operating System of each virtualized The traffic of admitted requests is, then, handled by the server. In this paper, we focus on the case where the CPU traffic controller and dispatcher. This component dispatches is the bottleneck resource of each server. This happens, for the requests over the VMs hosted by the processing nodes, example, when all applications running on the servers are while buffering the requests that are not immediately sent computationally intensive. Therefore, CPU frequency, switch to the VMs. Our data center supports multiple services that frequency and power and/or energy constraints are critical for can be requested by the clients, and each VM may host this scenario [13]. only one service; hence, queuing and dispatching operations According to [11], we consider a discrete-time model based take into account the deployment of VMs over the server on timeslots. Decisions on the capacity adjustment of the data infrastructure. Requests are distributed through the internal centers are taken at the beginning of each timeslot. We assume data center network to the appropriate processing servers (and that each timeslot is significantly longer than the job average VMs). A final element of our infrastructure is the internal interarrival time, so that resource provisioning may be based service of servers reconfiguration and consolidation. This on the average arrival rate during each slot. For example, service is in charge of the capacity adjustment, that decides the if requests arrive at a rate of few requests per seconds, the deployment of VMs over the servers and manages the servers timeslot can be in the order of minutes. by determining the ON/OFF status of the servers, as well as We now detail the formal model used to describe the system. the CPU frequencies by exploiting the DVFS capabilities of For the convenience of the readers, the major notations used in the underlying hardware. this paper are listed in Table I. In each slot t, new requests for The virtualized data center shown in Fig. 2 provides a i-th application arrive according to a random arrival process more detailed view of three main components: ADmission Ai (t) (IU/slot) with time average rate λi (IU/slot)1 . We Controller (ADC), Dispatcher, and Virtual Machine Monitor assume that the statistics of Ai (t) are unknown, so that (VMM). The proposed model is consistent with the architecture 1 The meaning of an Information Unit (IU) is application-dependent. We presented in [11]. We consider a separate ADC for each anticipate that, in the carried out tests of Section IV, IUs are understood as application. A set of dispatchers, one for each application, has M bit. 2016 IEEE Symposium on Computers and Communication (ISCC) " # TABLE I: Notation. X Wi (t + 1) = Wi (t) − aij Rij (t) + Ri (t), t ≥ 0, (1) Symbol Meaning/Role j + aij Indicator of the i-th application to the j-th server Ri (t) (IU/slot) Admitted arrival requests after ADC(i) in slot t Uij (t + 1) = [Uij (t) − µij (t)]+ + Rij (t), t ≥ 0, (2) Ai (t) (IU/slot) New arrival requests for i-th application in slot t Rij (t) (IU/slot) Routing decisions of i-th application to j-th server where Rij (t) is the number of requests for i-th application µij (t) (IU/slot) Service rate of i-th application on j-th server in slot t that are routed from its dispatcher to the j-th server in slot t. Wi (t) (IU/slot) i-th application buffer size in slot t Uij (t) (IU/slot) i-th application buffer size of the j-th server in slot t D. Computing and Switching RT Tj (s) Average round-trip-time of j-th dispatcher-server link fj (t) (IU/slot) Processing rate of j-th server in slot t The client requests are sent by the dispatchers to the active fjmax (IU/slot) Maximum allowed processing rate of j-th server VMs running on the servers for being processed. In this paper, rj (t)(IU/slot) Communication rate of the j-th server-dispatcher link Ecidle (j)(Joule) Static energy consumed by j-th server we assume that VMs deployed over a server can change their PN idle (j)(W att) Static power consumed by j-th dispatcher-server link share of server resources according to the model described Ecmax (j)(Joule) Maximum energy consumed by j-th server in [14], that is widely adopted in private cloud environments. Ec (j, t)(Joule) Computing energy consumed by j-th server in slot t Edyn (j, t)(Joule) Switch energy consumed by j-th server in slot t This model tends to face conditions of high computational ELAN (j, t)(Joule) Communication energy consumed by j-th server demand by means of few large VMs instead of many small Ejtot (t)(Joule) Total energy consumed by j-th server in slot t VMs. We adopt a simplified model where a single VM is {Ai (t) ∈ R+ deployed over each server and uses all the available resources 0 , t ≥ 0} is a random process with unknown statistics. for that server (we will refer to the VM deployed on server j as V M (j)). The extension of the model to the case where B. Admission Controller multiple VMs share the same server is left as an open issue For each application i ∈ A, the ADC of Fig. 2 decides to be addressed in future works. whether to admit or decline the new requests. The admitted In this paper, we model each server as capable of a process- requests are stored in the Dispatcher buffers (see the buffers ing rate fj (t) in slot t. Depending on the size Rij (t) (IU/slot) connected to each ADC), before being routed by the Dis- of the i-th application’s task to be currently processed by the patcher to one of the servers hosting that application. j-th server in slot t, the corresponding processing rate fj (t) We consider (time-slotted) G/G/1 fluid systems for mod- may be adaptively scaled at run-time through DVFS. In this eling the queue sets of ADC and servers of Fig. 2. Due to way, each server can be operated at multiple voltages which the admission control, both set of queues are loss-free and operate different CPU frequency [15]. Let Q be the number of they implement the FIFO service discipline. Specifically, at allowed CPU rates: for example, the Intel EIST-technologies the end of slot t, the arrival Ai (t), t ≥ 0 of processing new allows to choose the clock frequency from a set of Q = 62 (possibly, heterogeneous) requests arrive at the input of the or Q = 153 values for reducing the power consumption when ADCs of Fig. 2. Note that Ai (t) is assumed to be independent the computational demands are low. from the current backlogs of the ADC queues of Fig. 2. We DVFS is applied by the hosting physical servers to stretch assume that any new arrival that is not admitted by the ADC the processing times of the tasks and reduce energy consump- of Fig. 2 is declined. Let Ri (t) be the number of requests out tion by decreasing the processing rates of the active VMs. Each of Ai (t) that are admitted into the i-th application buffer of the processing rate may assume values over interval [0, fjmax ], Dispatcher by ADC(i). Let Wi (t) ∈ R+ 0 and Uij (t) ∈ R0 + where fjmax (IU/slot) is the maximum allowed processing be the numbers of IUs stored by ADCs and server queues of V M (j). of Fig. 2 at the beginning of slot t. Furthermore, let Rij In [16], it is shown that there is a quadratic relationship (IU/slot) be the size of task of the i-th application served between the CPU utilization of V M (j) and the corresponding by the j-th server that: (i) is drained from the ADC(i) queue computing energy consumption Ec (j, t) (all servers in our at the beginning of slot t; and, (ii) is injected into the ij- model are assumed to have identical CPU resources), so that th queue at the end of slot t. We assume that dispatching we can write: occurs on the basis of the VM-to-Server mapping provided by Ec (j, t) = Ecidle (j) + (fj (t)/fjmax )2 Ecmax (j) − Ecidle (j) (3) {aij }. If a given application is served by multiple VMs, we assume that the dispatcher uses the round-robin discipline for Furthermore, we note that switching from the processing rate the scheduling. fj (t−1) (e.g., the processing rate of V M (j) at (t−1)-th slot) to fj (t) (e.g., the processing rate of V M (j) in t-th slot) entails C. Adaptive Balancing Control and Dispatching an energy overhead of Edyn (j, t) [16]. Although the actual Let µij (t) (IU/slot) be the traffic that is drained from behavior of the function Edyn (j, t) depends on the adopted the j-th server queue of Fig. 2 during slot t. We denote the DVFS technique, a quite common model is the following buffer backlog by Wi (t), with 0 ≤ Wi (t) ≤ Ai (t). Hence, quadratic one [16]: the time evolutions of the backlogs {Wi (t) ∈ (R)+ 0 , t ≥ 0}, Edyn (j, t) = ke (fj (t) − fj (t − 1))2 . (4) {Uij (t) ∈ (R)+0 , t ≥ 0} of the ADC(i) and j-th server queues are dictated by the following Lindley’s equations, respectively: 2 http://download.intel.com/design/network/papers/30117401.pdf 3 https://software.intel.com/sites/default/files/ftalat.pdf 2016 IEEE Symposium on Computers and Communication (ISCC) In Eq. (4), ke (Joule/(Hz)2 ) denotes the switch cost induced Hence, the resulting total computing-plus-communication en- by an unit-size rate switching, which is typically limited up ergy Ejtot (t) consumed at slot t by the j-th server is: to few hundreds of µJ ′ s per (M Hz)2 [16]. Ejtot (t) = Ec (j, t) + Edyn (j, t) + ELAN (j, t). (9) At each slot, the VMM allocates the resources of each server among the VMs that host the applications running on III. P ROBLEM F ORMULATION AND S OLUTION ` be the set of turned ON VMs at slot t. that server. Let S(t) Hence, we have S(t)` ⊆ S. In our reference data center, the We now present our proposal of a dynamic load balancing scheduler interleaves reconfiguration and consolidation slots. and resource provisioning approach that takes into account Specifically, at each reconfiguration slot, the scheduler does computation and communication costs, and exploits DVFS- ` of turned ON VMs and, then, it leaves not change the set S(t) based discrete frequencies and their time shares for each VM. unchanged the corresponding sets of turned ON/OFF physical Our final goal is to maximize the throughput of the applica- servers. However, it assigns the requests Rij to the turned tions and minimize the joint computing-plus-communication ON VMs. Thus, the routing decisions Rij (t) must satisfy the energy costs of the servers, subject to the available control at slot t: following constraints ( options and to the structural constraints imposed by the con- ` sidered model. Since frequent turning ON/OFF servers may be 1, Rij = 0, if j ∈ / S(t) aij = ` (5) undesirable (for example, due to hardware reliability issues), 0, if j ∈ S(t); we will focus on frame-based control policies, where time X is divided into frames of length T slots. We define Tmax as 0≤ aij Rij (t) ≤ Wi (t) (6) ` the maximum number of slots considered in our experiments j∈S(t) for the evaluation of SLAs. We recall that the set of active E. Communication servers is chosen at the beginning of each frame and does not change for the duration of that frame (reconfiguration We assume that the j-th virtual end-to-end connection (e.g., state): this set may change in the next frame as workloads the j-th virtual link) of the data center of Fig. 2 is bidirec- change (consolidation state). In the consolidation state, we tional, symmetric and operates in a half-duplex way [17]. A use the MBFD [20] policy to migrate some of active servers’ joint analysis of the computing-plus-communication energy backlogs to some other servers, then we update the set of consumption in delay-tolerant data centers is performed in active servers. Let Ti and ξj denote the (average expected) rate [16], where the effects of inter networking infrastructures are of admitted requests for i-th application and the (average ex- evaluated. Two main conclusions arise from [16]. First, the pected) total joint computing-plus-communication Pt−1energy con- energy consumption due to the data transport may represent 1 sumption of j-th server, that is Ti , lim t−1 τ =0 E{Ri (τ )} a large part of the total energy consumption, especially at Pt−1 t←∞ 1 tot medium/high usage rates. Second, the energy consumption of and ξj , lim t−1 τ =0 E{Ej (τ )}, respectively. We define t←∞ cloud infrastructures needs to be analyzed by simultaneously T , { T1 , T2 , . . . , TN } as the vector of average rates of accounting for data computing and Dispatcher-Server data applications and ξ , { ξ1 , ξ2 , . . . , ξM } as the vector of energy transport. Motivated by these considerations, we consider the average rates of servers. Hence, the resource reconfigura- TCP New Reno protocol [18] to model the managed end- tion/consolidation problem at slot t is formulated as in the to-end Dispatcher-Server transport connections. Hence, under following: the Congestion Avoidance state, the power drained by each   connection may be evaluated as in [9]: X X 2 min θ ξj − βi Ti  , (10.1) Pjnet (t) = Ωj RT Tj rj (t) idle + PN (j), j = 1, . . . , M, (7) χ i∈A j∈S` where RT Tj is the average round-trip-time of the j-th server- subject to: dispatcher end-to-end connection, rj (t) is the communication 0 ≤ fj (t) ≤ fjmax , ` ∀j ∈ S, (10.2) rate of the j-th virtual link at the t-th slot, and Ωj is the j-th virtual link power coefficient that depends on the maximum 0 ≤ Ti ≤ λi , T ∈ Γ, ∀i ∈ A, (10.3) segment size of the transmitted packets and on the number of Eqs. (1), (2), (5), (6). (10.4) per-segment ACKs as described in [10], [19]. Hence, the corresponding one-way transmission delay is: ` In Eq.(10.1), where χ , {T , fj (t), Uij , Ri (t), i ∈ A, j ∈ S}. Dj (t) = (Rij (t))/rj (t), where Rij (t) is the workload of the θ and βi are non-negative weights which are used for normal- i-th application (requests) which is assigned to available j-th ization and to assign priorities between throughput and energy. server at the t-th slot. The overall two-way computing-plus- The optimization problem in Eq. (10.1) is a general weighted communication delay induced by the j-th end-to-end connec- linear combination of the sum throughput of the applications tion of Fig. 2 equates to 2Dj , so that the hard constraint on and the average energy usage in the data center. Eq. (10.2) the overall per-request execution time is: max1≤j≤M {2Dj }. (resp., Eq. (10.3)) bounds the maximum processing rate of j- Thus, the corresponding one-way communication energy th server (resp., maximum average rate of i-th application). At ELAN (j, t) wasted by the j-th virtual link at slot t is: the end, Γ in (10.3) represents the capacity region of the data ELAN (j, t) = Pjnet (t) (aij Rij (t)) /rj (t), ∀i ∈ A, (8) center, which is defined as the set of all possible longterm 2016 IEEE Symposium on Computers and Communication (ISCC) throughput values that can be achieved under any feasible Algorithm 1 A pseudo-code of the proposed JDLS resource allocation strategy. 1: for t ≥ 1 do In the rest of the paper, the optimal solution of the problem 2: if t 6= nT , then ⊲ t is a reconfiguration slot, 3: for all i ∈ A do ∗ (10) is denoted as in { fj∗ (t), Uij , Ri∗ (t), i ∈ A, j ∈ S }. We 4: Update Ri (t) using eq. (11); rely on the Lyapunov optimization [11], [21] to develop an 5: Update Rij (t) using eq. (5), (6) and JDLS’s Routing; optimal control policy. In particular, we introduce a dynamic 6: Update Wi (t + 1) using eq. (1); control algorithm that achieves the optimal solution {Ti∗ } and 7: end for {ξj∗ }. We use queue backlog values in slot t to make decisions 8: for all j ∈ S` do 9: Update µij (t) using eq. (12); for the proposed algorithm in ADC, Routing and VMM 10: Update Uij (t + 1) using eq. (2); components of Fig. 2. These decisions help us to dynamically 11: end for update the system condition, while the new requests come 12: else ⊲ t is a consolidation slot to the system. Hence, the proposed algorithm optimizes the 13: re-run lines 3-7; objective in (10) by solving a sequence of optimization prob- 14: for all j ∈ S` do 15: Update µij (t) using eq. (12) and update eq. (3), (4), lems over time. The queue backlogs can be viewed as dynamic and (8) with zero frequency; Lagrange multipliers that enable stochastic optimization [21]. 16: Request task replacement using MBFD [20] policy The proposed method called Joint Dynamic Lyapunov-based ` over S(t); Scheduler (JDLS) performs the following actions: 17: ` Update S(t); 1) Admission control: it is important to maximize Ri (t) 18: end for 19: end if and minimize the application backlog Wi (t) for i-th 20: end for application in slot t. So, we should solve the following problem: min Ri (t) [Wi (t) − V βi ] , (11.1) IV. T EST R ESULTS AND P ERFORMANCE C OMPARISONS i s.t: 0 ≤ Ri (t) ≤ Ai (t), (11.2) This section presents the performance evaluation of the proposed scheduler under a set of synthetic and real-world where V ≥ 0 is a control parameter used by the input traffic traces. In particular, we perform a sensitivity system administrator to control the trade-off between analysis with respect to the main model parameters; then, average delay and total average utility by exploiting the we compare the performance of the proposed JDLS scheduler threshold-based mechanism used in [11]. If the current with that of the recent GRADient-based Iterative Scheduler ADC queue backlog for i-th application satisfies the (GRADIS) [10], the Static Lyapunov-based Scheduler (SLS) condition Wi (t)>V βi , then Ri∗ (t) = 0 and no new in [11], the hybrid NetDC scheduler (H-NetDC) in [12], and requests are admitted. Otherwise, Ri∗ (t) = Ai (t) and the NetDC scheduler (NetDC) in [9]. It is worth to note that the all new requests are admitted. parameters of each scheduler have been tuned in preliminary 2) Request dispatching: for the i-th application, we dispatch experiments to ensure a fair comparison among the considered the ADC(i)’s drained queue requests to the active ` using a round-robin policy. approaches. As metrics for the scheduler performance evalua- servers in set S(t) tion, we consider the total utility (Throughput) Utot , the total 3) VMM and resource allocation: for j-th server in the set ∗ ∗ ` consumed energy ξ j and the total delay T tot of the admitted S(t), we have X to solve the following sub-problem: requests. min V θEjtot (t) − Uij (t)µij (t), (12.1) i i∈A A. Simulated Setup s.t: Eqs. (3), (4), (7), (12.2) The simulations have been carried out by exploiting the nu- Eq. (12) is a generalized min-weight problem, where the merical software of the MATLAB platform. They emulate 10 service rate provided to any application is weighted by quad-core Dell PowerEdge servers, equipped with 3.06 GHz its current queue backlog. Hence, the optimal solution Intel Xeon CPU and 4GB of RAM. All the emulated servers allocates resources to maximize the are connected through commodity Fast Ethernet NICs. In all P service rate of the most backlogged application: i∈A Uij (t)µij (t), carried out tests, we configure the VMs with 512MB of RAM and minimize the overall energy consumption for j- and emulate the TCP New Reno protocol for implementing tot the needed VM-to-VM transport connections. P th server: i∈A V θE j (t). It is worth to note that each server solves its own resource allocation problem For the experiments, we consider two different scenarios. independently by using the queue backlog values of the The first scenario includes a synthetic workload, whose main applications hosted on it and this can be implemented parameters are detailed in Table II. The second scenario refers in a fully distributed way, with a major benefit in terms to a real-world workload trace, which is represented in Fig. 3: of scalability. this is the same real-world workload trace considered in [22]. Algorithm 1 reports a pseudo-code for the adaptive imple- We perform preliminary experiments and we found that the mentation of the proposed resource scheduler. best parameter values for the real-world workload are ke = 0.5 [Joule/(M Hz)2 ] and T = 1.2 [s]. 2016 IEEE Symposium on Computers and Communication (ISCC) TABLE II: Default values of the main simulated parameters. 1000 200 Parameter Value ∗ 800 T tot 150 (∆, γ = θ/βi ) (1 (s), {0.5, 1, 2}) Utot , γ = 2 (T, Tmax ) (100, 1000) Utot , γ = 1 idle 600 Utot , γ = 0.5 100 (Ωj , PN (j)) (0.5, 0.5) (W att) ∀j ∈ S T tot Utot (M, N, V ) ∗ (100, 10,{0:500:20000}) Ai 8 (M bit) ∀i ∈ A 400 50 PMR4 2 ke , Q 0.05 (J/(M Hz)2 ), 6 rjmax , fjmax 100, 10 (M bit/s) ∀j ∈ S 200 0 RT Tj 70 (µs) ∀j ∈ S (µmin ij , µij max ) {(200, 400), (50, 300)} (M bit/slot) 0 −50 0 0.5 1 1.5 2 (Ecidle (j), Ecmax (j)) (5, 240)(Joule) ∀j ∈ S 4 V x 10 (a) Linear µij 1000 200 16 ∗ 14 800 T tot 150 Utot , γ = 2 Number of arrivals per slot 12 Utot , γ = 1 600 Utot , γ = 0.5 100 T tot Utot 10 ∗ 8 400 50 6 200 0 4 2 0 −50 0 10 20 30 40 50 60 0 0.5 1 1.5 2 Slot index 4 V x 10 Fig. 3: Measured workload trace: PMR = 1.526. (b) Quadratic µij ∗ Fig. 4: T tot vs. V and Utot for the synthetic workload for B. Experimental Results linear (4a) and quadratic (4b) control decision µij . In order to measure the performance of the proposed sched- uler JDLS, we perform different experiments detailed in the solves its own resource allocation problem independently by following subsections. using the queue backlog values of the applications hosted on it 1) Throughput and Average Total Delay: In this experiment and this allows, in return, the implementation of the proposed we evaluate the total average utility Utot and the average total approach in a fully distributed way. ∗ delay of the admitted requests T tot for different values of the 2) Sensitivity to the data center parameters: Fig. 5 reports ∗ ∗ input parameters V and γ. We define T tot like [23] as the the average total consumed energy ξ j for servers for various sum of the delay in the first queue blocks (ADC buffers) and values of M and other parameters, such as Q (Fig. 5a), ke second queue blocks (VM buffers), expressed as the number (Fig. 5b), and Ωj (Fig. 5c). Specifically, Fig. 5 points out that: ∗ of time slots. Fig. 4a and 4b show the measured Utot and (i) while M increases, the ξ j decreases; (ii) while Q, ke , and ∗ T tot under the linear and quadratic control decisions µij of ∗ Ωj increases, the ξ j declines according to the Eqs. (3), (4) and Eq. (12), respectively. (8), respectively. In Fig. 5a, Q = ∞ indicates the case of real- The observation of these results leads to two main conclu- time results where we have no queues: in this case, the optimal sions. First, Utot (which is the solution of (10.1)) decreases frequency of each server is the corresponding frequency that for increasing values of the weight factor of γ , θ/βi , ∀i and we need for the incoming workload (ideal case). ∗ V . Utot converges to a minimum value, and T tot increases 3) Performance Comparisons: We now present a compar- and converges to a maximum value for larger values of V . ison of the proposed JDLS scheduler with other available This is due to the fact that, according to the conditional alternatives in terms of consumed energy, total delay and Lyapunov optimization approach in [24] (exploiting Lyapunov execution times. ∗ drift theorem), the performance of the minimization algorithm Fig. 6 presents the average total consumed energy ξ j and is bounded up to a finite constant which depends on the arrival ∗ the average total delay T tot versus the number of VMs M for and service rates [24]. Second, we conclude that the average the aforementioned schedulers [9]–[12] for the synthetic work- overall time and utility of the presented data center is inde- pendent of the control decision µij . It means that each server 4 PMR:=Peak-to-Mean Ratio of the input workload 2016 IEEE Symposium on Computers and Communication (ISCC) 300 100 300 Q=2 ke = 5 Ωj = 0.2 Q=6 ke = 50 Ωj = 0.5 80 Q = 15 ke = 500 200 Ωj = 0.2 + 0.125(j − 1) 200 Q=∞ Ωj = 0.5 + 0.25(j − 1) 60 ∗ ξj ∗ ξj ∗ ξj 100 40 100 20 0 10 0 5 10 15 20 10 12 14 4 6 8 10 12 14 16 18 20 M M 16 18 20 M (a) Q (b) ke (c) Ωj Fig. 5: Sensitivity analysis with respect to parameters Q (5a), ke (5b) and Ωj at V = 104 , ke = 5 (5c). load. In detail, Fig. 6a shows that the average energy saving 220 of the JDLS scheduler over NetDC, GRADIS, SLS H-NetDC methods is about 75%, 70%, 85% and 91%, respectively. This result confirms that the proposed JDLS scheduler is capable 100 to adapt to the incoming behaviors of the input workload by 60 increasing the number of turned ON VMs faster than other ∗ X: 55 ∗ Y: 54.03 ξj methods. Fig. 6b compares the average total delay T tot of JDLS JDLS, GRADIS [10] and SLS [11] for increasing values of GRADIS M . In this case we do not consider the NetDC and H-NetDC N etDC SLS schedulers because they are real-time approaches which do H-N etDC not use any queue for the incoming requests. The results of 15 Fig. 6b show that the average total saving of JDLS over the 10 GRADIS is almost 30%, but is lower with respect to SLS; it is 10 20 30 40 50 60 70 80 90 100 M important to consider that JDLS performs communication joint ∗ (a) ξj -vs.-M switching optimization whilst SLS does not perform them. 700 We also evaluate how JDLS can handle a real world 600 scenario. To this aim, we consider the traces in Fig. 3 [22] pre- viously described. Fig. 7 compares the average total consumed ∗ energy ξ j for different values of M for the aforementioned 400 schedulers [9]–[12]. These results show that by increasing T tot ∗ the number of servers, the average energy consumption per- J DLS SLS server decreases for all the tested schedulers. The amount of GRADIS energy saving of JDLS compared to SLS, NetDC, GRADIS, 200 and H-NetDC is about 30%, 50%, 51% and 78%, respectively. This last comparison confirms that JDLS can reduce energy consumption even in the highly variable scenario that charac- 0 10 20 30 40 50 60 70 80 90 100 terizes real world workloads. M ∗ In the last simulation, we evaluate the average execution (b) T tot -vs.-M time for each of the five considered schedulers for differ- Fig. 6: Performance comparison under synthetic workload ent number of slots (Tmax = 103 , 104 ) for the synthetic (with Tmax = 104 and V = 104 ). and the real-world workload traces. The results reported in 104 Table III show that the JDLS scheduler is able to work with a significantly lower average execution time with respect to the GRADIS and the NetDC/H-NetDC methods, in both 103 JDLS scenarios and scales of incoming workloads. The NetDC/H- GRADIS ∗ NetDC ξj NetDC methods are considered together because, from a SLS H-NetDC computational point of view, they perform the same operations 102 and therefore have the same speed. As shown in Table III, the average time saving of the JDLS scheduler for the higher amount of time slots (Tmax = 104 ) over the GRADIS and 101 10 20 30 40 50 60 70 80 90 100 M the NetDC/H-NetDC alternatives is of about 48% and 94.7% ∗ in the synthetic workload, and of 59% and 66% in the real- Fig. 7: T tot comparison under real workload. world workload, respectively. On the other hand, with respect to the SLS scheduler, the JDLS execution time is about 3% incoming workload (Tmax = 104 ). It is important to note (0.9 ms) and 13% (5.5 ms) higher in the same scale of that, differently from the SLS scheduler, the JDLS proposal 2016 IEEE Symposium on Computers and Communication (ISCC) TABLE III: Average execution time (ms) of considered sched- ulers for synthetic and real-world workloads. [2] D. Ardagna, M. Ciavotta, and R. Lancellotti, “A receding horizon approach for the runtime management of iaas cloud systems,” in 16th Workload JDLS SLS GRADIS NetDC/ IEEE SYNASC, 2014, pp. 445–452. H-NetDC [3] C. Canali and R. Lancellotti, “Exploiting classes of virtual machines for Real-world (Tmax = 104 ) 26.7 25.8 65.3 78.6 scalable iaas cloud management,” in Proc. of the 4th NCCA, 2015, pp. Real-world (Tmax = 103 ) 3.6 2.9 7 7.3 15–22. Synthetic (Tmax = 104 ) 44.1 38.6 85.1 840 [4] R. W. Ahmad, A. Gani, S. H. A. Hamid, M. Shiraz, A. Yousafzai, and Synthetic (Tmax = 103 ) 4 3.4 8 84.8 F. Xia, “A survey on virtual machine migration and server consolidation frameworks for cloud data centers,” Journal of Network and Computer Applications, vol. 52, pp. 11–25, 2015. includes the communication and switch components in the [5] D. Ardagna, B. Panicucci, M. Trubian, and L. Zhang, “Energy-aware model: even with this additional element, the execution time autonomic resource allocation in multitier virtualized environments,” of JDLS is comparable with the SLS alternative. Services Computing, IEEE Transactions on, vol. 5, no. 1, pp. 2–19, To summarize our results, we can conclude that the JDLS Jan 2012. [6] L. Chiaraviglio, D. Ciullo, M. Mellia, and M. Meo, “Modeling sleep scheduler can significantly outperform every other considered mode gains in energy-aware networks,” Computer Networks, vol. 57, solution in reducing the computing-plus-communication en- no. 15, pp. 3051–3066, 2013. ergy for a wide range of workloads (including real-world [7] N. Cordeschi, T. Patriarca, and E. Baccarelli, “Stochastic traffic engi- traces) and parameter setups. Furthermore, the scheduler is neering for real-time applications over wireless networks,” Journal of Network and Computer Applications, vol. 35, no. 2, pp. 681–694, 2012. extremely fast in taking decisions, suggesting that it has [8] C. Wilson, H. Ballani, T. Karagiannis, and A. Rowtron, “Better never potential for scalability over large data centers and large input than late: Meeting deadlines in datacenter networks,” in ACM SIG- workloads. COMM, vol. 41, no. 4, 2011, pp. 50–61. [9] N. Cordeschi, M. Shojafar, and E. Baccarelli, “Energy-saving self- V. C ONCLUSIONS AND F UTURE W ORK configuring networked data centers,” Computer Networks, vol. 57, no. 17, pp. 3479–3491, 2013. In this paper, we proposed an effective dynamic scheduler [10] M. Shojafar, C. Nicola, and E. Baccarelli, “Energy-efficient adaptive for the joint adaptive tuning of the: (i) admitted traffic; resource management for real-time vehicular cloud services,” IEEE (ii) delivered throughput; and (iii) resource reconfiguration Transactions on Cloud Computing (TCC), vol. PP, no. 99, p. 1, 2016. and consolidation of virtualized networked data center plat- [11] R. Urgaonkar, U. C. Kozat, K. Igarashi, and M. J. Neely, “Dynamic resource allocation and power management in virtualized data centers,” forms. The overall goal is to reduce the energy consumption in Proc. of IEEE/IFIP NOMS, 2010, pp. 479–486. while guaranteeing QoS in cloud scenarios characterized by [12] N. Cordeschi, D. Amendola, F. De Rango, and E. Baccarelli, high computational demand and delay sensitivity. Remarkable “Networking-computing resource allocation for hard real-time green cloud applications,” in Wireless Days, 2014, pp. 1–4. features of the developed joint scheduler are that: (i) its [13] A. Mishra, R. Jain, and A. Durresi, “Cloud computing: networking and implementation is distributed and adaptive, and the resulting communication challenges,” IEEE Communications Magazine, vol. 50, complexity scales with the number of the available VMs; (ii) no. 9, pp. 24–25, 2012. it minimizes the energy consumed by the overall platform for [14] D. Gmach, J. Rolia, and L. Cherkasova, “Selling t-shirts and time shares in the cloud,” in Proc. of 12th IEEE/ACM CCGrid, 2012, pp. 539–546. computing, router-server communication and server reconfigu- [15] S. Azodolmolky, P. Wieder, and R. Yahyapour, “Cloud computing ration; and, (iii) despite the unpredictable time-varying nature networking: challenges and opportunities for innovations,” IEEE Com- of the input workload, it is capable to provide QoS guarantees, munications Magazine, vol. 51, no. 7, pp. 54–62, 2013. in terms of maximizing delivered average throughput, and [16] M. Portnoy, Virtualization essentials. John Wiley & Sons, 2012. maximum queuing-plus-computing delay. Actual performance [17] J. Baliga, R. W. Ayre, K. Hinton, and R. Tucker, “Green cloud comput- ing: Balancing energy in processing, storage, and transport,” Proceedings of the proposed scheduler has been numerically tested under of the, IEEE, vol. 99, no. 1, pp. 149–167, 2011. both synthetic and real-world traces for the input traffic under [18] O. Tamm, C. Hermsmeyer, and A. M. Rush, “Eco-sustainable system and various conditions, allowing us to drive analytical performance network architectures for future transport networks,” Bell Labs Technical guarantees of the algorithm. This work can be extended in Journal, vol. 14, no. 4, pp. 311–327, 2010. [19] B. Kantarci and H. T. Mouftah, “Delay-constrained admission and some directions of potential interest. In particular, we can bandwidth allocation for long-reach epon,” Journal of Networks, vol. 7, enrich the application scenario to consider intra-slot traffic no. 5, pp. 812–820, 2012. arrivals and we can introduce live migration of VMs to achieve [20] A. Beloglazov, J. Abawajy, and R. Buyya, “Energy-aware resource additional energy savings. allocation heuristics for efficient management of data centers for cloud computing,” Future generation computer systems, vol. 28, no. 5, pp. VI. ACKNOWLEDGEMENT 755–768, 2012. [21] L. Georgiadis, M. J. Neely, and L. Tassiulas, Resource allocation and The first three authors acknowledge the support of the cross-layer control in wireless networks. Now Publishers Inc, 2006. University of Modena and Reggio Emilia through the project [22] B. Urgaonkar, G. Pacifici, P. Shenoy, M. Spreitzer, and A. Tantawi, “An- SAMMClouds: Secure and Adaptive Management of Multi- alytic modeling of multitier internet applications,” ACM Transactions on the Web (TWEB), vol. 1, no. 1, p. 2, 2007. Clouds. [23] M. Dabbagh, N. Sayegh, A. Kayssi, I. Elhajj, and A. Chehab, “Fast dy- namic internet mapping,” Future Generation Computer Systems, vol. 39, R EFERENCES pp. 55–66, 2014. [1] J. Whitney and P. Delforge, “Data center efficiency assessment – scaling [24] M. J. Neely, “Stochastic network optimization with application to up energy efficiency across the data center industry: Evaluating key communication and queueing systems,” Synthesis Lectures on Commu- drivers and barriers,” NRDC, Anthesis, Tech. Rep., 2014. nication Networks, vol. 3, no. 1, pp. 1–211, 2010.