Package org.apache.storm.scheduler
Class Cluster
java.lang.Object
org.apache.storm.scheduler.Cluster
- All Implemented Interfaces:
ISchedulingState
- Direct Known Subclasses:
SingleTopologyCluster
The current state of the storm cluster. Cluster is not currently thread safe.
-
Constructor Summary
ConstructorsConstructorDescriptionCopy constructor.Cluster(Cluster src, Topologies topologies) Testing Constructor that takes an existing cluster and replaces the topologies in it.Cluster(INimbus nimbus, ResourceMetrics resourceMetrics, Map<String, SupervisorDetails> supervisors, Map<String, ? extends SchedulerAssignment> assignments, Topologies topologies, Map<String, Object> conf) -
Method Summary
Modifier and TypeMethodDescriptionprotected voidassertValidTopologyForModification(String topologyId) Check if the given topology is allowed for modification right now.voidassign(SchedulerAssignment assignment, boolean ignoreSingleExceptions) Assign everything for the given topology.voidassign(WorkerSlot slot, String topologyId, Collection<ExecutorDetails> executors) Assign the slot to the executors for this topology.voidblacklistHost(String host) voidfreeSlot(WorkerSlot slot) Free the specified slot.voidfreeSlots(Collection<WorkerSlot> slots) free the slots.Get all scheduled resources for node.getAssignablePorts(SupervisorDetails supervisor) Get the ports that are not blacklisted.Get all non-blacklisted slots in the cluster.getAssignableSlots(SupervisorDetails supervisor) Return all non-blacklisted slots on this supervisor.static doublegetAssignedMemoryForSlot(Map<String, Object> topConf) Get heap memory usage for a worker's main process and logwriter process.intgetAssignedNumWorkers(TopologyDetails topology) Get the number of workers assigned to a topology.getAssignmentById(String topologyId) get the current assignment for the topology.Get all the assignments.getAvailablePorts(SupervisorDetails supervisor) Return the available ports of this supervisor.Get the resources on the supervisor that are available to be scheduled.Get all the available worker slots in the cluster.getAvailableSlots(SupervisorDetails supervisor) Return all the available slots on this supervisor.Get all of the hosts that are blacklisted.doubleGet the total amount of CPU resources in cluster.Get the total amount of generic resources (excluding CPU and memory) in cluster.doubleGet the total amount of memory resources in cluster.getConf()Get the nimbus configuration.Map a supervisor to a given host.doubleGet the component name to executor list for executors that need to be scheduled.Get the executor to component name map for executors that need to be scheduled.Get the network topography (rackId -> nodes in the rack).getNonBlacklistedAvailableSlots(List<String> blacklistedSupervisorIds) Get all the available worker slots in the cluster, that are not blacklisted.getNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds) Get the resources in the cluster that are available for scheduling.doublegetScheduledCpuForNode(String nodeId) Get the total cpu currently scheduled on a node.doublegetScheduledMemoryForNode(String nodeId) Get the total memory currently scheduled on a node.Get all topology scheduler statuses.getSupervisorById(String nodeId) Get a specific supervisor with thenodeId.Get all the supervisors.getSupervisorsByHost(String host) Get all the supervisors on the specifiedhost.Get the amount of used and free resources on a supervisor.Get all of the topologies.Get the amount of resources used by topologies.getUnassignedExecutors(TopologyDetails topology) get the unassigned executors of the topology.getUsedPorts(SupervisorDetails supervisor) Get all the used ports of this supervisor.Get all currently occupied slots.getUsedSlotsByTopologyId(String topologyId) get slots used by a topology.Get the resources for a given slot.Gets the reference to the full topology->worker resource map.booleanisBlackListed(String supervisorId) Check is a given supervisor is on a blacklisted host.booleanisBlacklistedHost(String host) Check if a given host is blacklisted.booleanisSlotOccupied(WorkerSlot slot) Check if a slot is occupied or not.booleanneedsScheduling(TopologyDetails topology) Does the topology need scheduling.booleanneedsSchedulingRas(TopologyDetails topology) LikeISchedulingState.needsScheduling(TopologyDetails)but does not take into account the number of workers requested.Get all of the topologies that need scheduling.voidsetAssignments(Map<String, ? extends SchedulerAssignment> newAssignments, boolean ignoreSingleExceptions) Set assignments for cluster.voidsetBlacklistedHosts(Set<String> hosts) Set the list of hosts that are blacklisted.voidsetGreyListedSupervisors(Set<String> greyListedSupervisors) voidsetNetworkTopography(Map<String, List<String>> networkTopography) voidset scheduler status for a topology.voidsetStatus(TopologyDetails td, String statusMessage) set scheduler status for a topology.voidsetStatusIfAbsent(String topologyId, String statusMessage) voidsetStatusMap(Map<String, String> statusMap) set scheduler status map.voidUnassign everything for the given topology id.voidupdateFrom(Cluster other) Update the assignments and status from the other cluster.booleanwouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap) Would scheduling exec on ws fit?Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface org.apache.storm.scheduler.ISchedulingState
getAssignedRacks, getHostToRack
-
Constructor Details
-
Cluster
public Cluster(INimbus nimbus, ResourceMetrics resourceMetrics, Map<String, SupervisorDetails> supervisors, Map<String, ? extends SchedulerAssignment> assignments, Topologies topologies, Map<String, Object> conf) -
Cluster
Copy constructor. -
Cluster
Testing Constructor that takes an existing cluster and replaces the topologies in it.- Parameters:
src- the original clustertopologies- the new topolgoies to use
-
-
Method Details
-
getAssignedMemoryForSlot
Get heap memory usage for a worker's main process and logwriter process.- Parameters:
topConf- - the topology config- Returns:
- the assigned memory (in MB)
-
assertValidTopologyForModification
Check if the given topology is allowed for modification right now. If not throw an IllegalArgumentException else go on.- Parameters:
topologyId- the id of the topology to check
-
getTopologies
Description copied from interface:ISchedulingStateGet all of the topologies.- Specified by:
getTopologiesin interfaceISchedulingState- Returns:
- all of the topologies that are a part of the cluster.
-
getBlacklistedHosts
Description copied from interface:ISchedulingStateGet all of the hosts that are blacklisted.- Specified by:
getBlacklistedHostsin interfaceISchedulingState- Returns:
- all of the hosts that are blacklisted
-
setBlacklistedHosts
Set the list of hosts that are blacklisted.- Parameters:
hosts- the new hosts that are blacklisted.
-
blacklistHost
-
isBlackListed
Description copied from interface:ISchedulingStateCheck is a given supervisor is on a blacklisted host.- Specified by:
isBlackListedin interfaceISchedulingState- Parameters:
supervisorId- the id of the supervisor- Returns:
- true if it is else false
-
isBlacklistedHost
Description copied from interface:ISchedulingStateCheck if a given host is blacklisted.- Specified by:
isBlacklistedHostin interfaceISchedulingState- Parameters:
host- the name of the host- Returns:
- true if it is else false.
-
getHost
Description copied from interface:ISchedulingStateMap a supervisor to a given host.- Specified by:
getHostin interfaceISchedulingState- Parameters:
supervisorId- the id of the supervisor- Returns:
- the actual host name the supervisor is on
-
needsSchedulingTopologies
Description copied from interface:ISchedulingStateGet all of the topologies that need scheduling.- Specified by:
needsSchedulingTopologiesin interfaceISchedulingState- Returns:
- all of the topologies that are not fully scheduled.
-
needsScheduling
Description copied from interface:ISchedulingStateDoes the topology need scheduling.A topology needs scheduling if one of the following conditions holds:
- Although the topology is assigned slots, but is squeezed. i.e. the topology is assigned less slots than desired.
- There are unassigned executors in this topology
- Specified by:
needsSchedulingin interfaceISchedulingState
-
needsSchedulingRas
Description copied from interface:ISchedulingStateLikeISchedulingState.needsScheduling(TopologyDetails)but does not take into account the number of workers requested. This is because the number of workers is ignored in RAS- Specified by:
needsSchedulingRasin interfaceISchedulingState- Parameters:
topology- the topology to check- Returns:
- true if the topology needs scheduling else false.
-
getNeedsSchedulingExecutorToComponents
Description copied from interface:ISchedulingStateGet the executor to component name map for executors that need to be scheduled.- Specified by:
getNeedsSchedulingExecutorToComponentsin interfaceISchedulingState- Parameters:
topology- the topology this is for- Returns:
- a executor -> component-id map which needs scheduling in this topology.
-
getNeedsSchedulingComponentToExecutors
public Map<String,List<ExecutorDetails>> getNeedsSchedulingComponentToExecutors(TopologyDetails topology) Description copied from interface:ISchedulingStateGet the component name to executor list for executors that need to be scheduled.- Specified by:
getNeedsSchedulingComponentToExecutorsin interfaceISchedulingState- Parameters:
topology- the topology this is for- Returns:
- a component-id -> executors map which needs scheduling in this topology.
-
getUsedPorts
Description copied from interface:ISchedulingStateGet all the used ports of this supervisor.- Specified by:
getUsedPortsin interfaceISchedulingState
-
getAvailablePorts
Description copied from interface:ISchedulingStateReturn the available ports of this supervisor.- Specified by:
getAvailablePortsin interfaceISchedulingState
-
getAssignablePorts
Description copied from interface:ISchedulingStateGet the ports that are not blacklisted.- Specified by:
getAssignablePortsin interfaceISchedulingState- Parameters:
supervisor- the supervisor- Returns:
- the ports that are not blacklisted
-
getNonBlacklistedAvailableSlots
Description copied from interface:ISchedulingStateGet all the available worker slots in the cluster, that are not blacklisted.- Specified by:
getNonBlacklistedAvailableSlotsin interfaceISchedulingState- Parameters:
blacklistedSupervisorIds- list of supervisor ids that should also be considered blacklisted.
-
getAvailableSlots
Description copied from interface:ISchedulingStateGet all the available worker slots in the cluster.- Specified by:
getAvailableSlotsin interfaceISchedulingState
-
getAvailableSlots
Description copied from interface:ISchedulingStateReturn all the available slots on this supervisor.- Specified by:
getAvailableSlotsin interfaceISchedulingState
-
getAssignableSlots
Description copied from interface:ISchedulingStateReturn all non-blacklisted slots on this supervisor.- Specified by:
getAssignableSlotsin interfaceISchedulingState- Parameters:
supervisor- the supervisor- Returns:
- the non-blacklisted slots
-
getAssignableSlots
Description copied from interface:ISchedulingStateGet all non-blacklisted slots in the cluster.- Specified by:
getAssignableSlotsin interfaceISchedulingState
-
getUnassignedExecutors
Description copied from interface:ISchedulingStateget the unassigned executors of the topology.- Specified by:
getUnassignedExecutorsin interfaceISchedulingState- Parameters:
topology- the topology to check- Returns:
- the unassigned executors of the topology.
-
getAssignedNumWorkers
Description copied from interface:ISchedulingStateGet the number of workers assigned to a topology.- Specified by:
getAssignedNumWorkersin interfaceISchedulingState- Parameters:
topology- the topology this is for- Returns:
- the number of workers assigned to this topology.
-
getAvailableResources
Description copied from interface:ISchedulingStateGet the resources on the supervisor that are available to be scheduled.- Specified by:
getAvailableResourcesin interfaceISchedulingState- Parameters:
sd- the supervisor.- Returns:
- the resources available to be scheduled.
-
wouldFit
public boolean wouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap) Description copied from interface:ISchedulingStateWould scheduling exec on ws fit? With a heap <= maxHeap total memory added <= memoryAvailable and cpu added <= cpuAvailable.- Specified by:
wouldFitin interfaceISchedulingState- Parameters:
ws- the slot to put it inexec- the executor to investigatetd- the topology detains for this executorresourcesAvailable- all the available resourcesmaxHeap- the maximum heap size for ws- Returns:
- true it fits else false
-
assign
Assign the slot to the executors for this topology.- Throws:
RuntimeException- if the specified slot is already occupied.
-
assign
Assign everything for the given topology.- Parameters:
assignment- the new assignment to make
-
freeSlot
Free the specified slot.- Parameters:
slot- the slot to free
-
freeSlots
free the slots.- Parameters:
slots- multiple slots to free
-
isSlotOccupied
Description copied from interface:ISchedulingStateCheck if a slot is occupied or not.- Specified by:
isSlotOccupiedin interfaceISchedulingState- Parameters:
slot- the slot be to checked.- Returns:
- true if the specified slot is occupied.
-
getAssignmentById
Description copied from interface:ISchedulingStateget the current assignment for the topology.- Specified by:
getAssignmentByIdin interfaceISchedulingState
-
getUsedSlotsByTopologyId
Description copied from interface:ISchedulingStateget slots used by a topology.- Specified by:
getUsedSlotsByTopologyIdin interfaceISchedulingState
-
getSupervisorById
Description copied from interface:ISchedulingStateGet a specific supervisor with thenodeId.- Specified by:
getSupervisorByIdin interfaceISchedulingState
-
getUsedSlots
Description copied from interface:ISchedulingStateGet all currently occupied slots.- Specified by:
getUsedSlotsin interfaceISchedulingState
-
getSupervisorsByHost
Description copied from interface:ISchedulingStateGet all the supervisors on the specifiedhost.- Specified by:
getSupervisorsByHostin interfaceISchedulingState- Parameters:
host- hostname of the supervisor- Returns:
- the
SupervisorDetailsobject.
-
getAssignments
Description copied from interface:ISchedulingStateGet all the assignments.- Specified by:
getAssignmentsin interfaceISchedulingState
-
setAssignments
public void setAssignments(Map<String, ? extends SchedulerAssignment> newAssignments, boolean ignoreSingleExceptions) Set assignments for cluster. -
getSupervisors
Description copied from interface:ISchedulingStateGet all the supervisors.- Specified by:
getSupervisorsin interfaceISchedulingState
-
getNonBlacklistedClusterAvailableResources
public NormalizedResourceOffer getNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds) Description copied from interface:ISchedulingStateGet the resources in the cluster that are available for scheduling.- Specified by:
getNonBlacklistedClusterAvailableResourcesin interfaceISchedulingState- Parameters:
blacklistedSupervisorIds- other ids that are tentatively blacklisted.
-
getClusterTotalCpuResource
public double getClusterTotalCpuResource()Description copied from interface:ISchedulingStateGet the total amount of CPU resources in cluster.- Specified by:
getClusterTotalCpuResourcein interfaceISchedulingState
-
getClusterTotalMemoryResource
public double getClusterTotalMemoryResource()Description copied from interface:ISchedulingStateGet the total amount of memory resources in cluster.- Specified by:
getClusterTotalMemoryResourcein interfaceISchedulingState
-
getClusterTotalGenericResources
Description copied from interface:ISchedulingStateGet the total amount of generic resources (excluding CPU and memory) in cluster.- Specified by:
getClusterTotalGenericResourcesin interfaceISchedulingState
-
getNetworkTopography
Description copied from interface:ISchedulingStateGet the network topography (rackId -> nodes in the rack).- Specified by:
getNetworkTopographyin interfaceISchedulingState
-
setNetworkTopography
-
setStatus
set scheduler status for a topology. -
setStatus
set scheduler status for a topology. -
setStatusIfAbsent
-
getStatusMap
Description copied from interface:ISchedulingStateGet all topology scheduler statuses.- Specified by:
getStatusMapin interfaceISchedulingState
-
setStatusMap
set scheduler status map. -
getStatus
-
getTopologyResourcesMap
Description copied from interface:ISchedulingStateGet the amount of resources used by topologies. Used for displaying resource information on the UI.- Specified by:
getTopologyResourcesMapin interfaceISchedulingState- Returns:
- a map that contains multiple topologies and the resources the topology requested and assigned. Key: topology id Value: an array that describes the resources the topology requested and assigned in the following format: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
-
getSupervisorsResourcesMap
Description copied from interface:ISchedulingStateGet the amount of used and free resources on a supervisor. Used for displaying resource information on the UI- Specified by:
getSupervisorsResourcesMapin interfaceISchedulingState- Returns:
- a map where the key is the supervisor id and the value is a map that represents resource usage for a supervisor in the following format: {totalMem, totalCpu, usedMem, usedCpu}
-
getWorkerResourcesMap
Description copied from interface:ISchedulingStateGets the reference to the full topology->worker resource map.- Specified by:
getWorkerResourcesMapin interfaceISchedulingState- Returns:
- map of topology -> map of worker slot ->resources for that worker
-
getWorkerResources
Description copied from interface:ISchedulingStateGet the resources for a given slot.- Specified by:
getWorkerResourcesin interfaceISchedulingState- Parameters:
ws- the slot- Returns:
- the resources currently assigned
-
getResourceMetrics
-
getAllScheduledResourcesForNode
Description copied from interface:ISchedulingStateGet all scheduled resources for node.- Specified by:
getAllScheduledResourcesForNodein interfaceISchedulingState
-
getScheduledMemoryForNode
Description copied from interface:ISchedulingStateGet the total memory currently scheduled on a node.- Specified by:
getScheduledMemoryForNodein interfaceISchedulingState- Parameters:
nodeId- the id of the node- Returns:
- the total memory currently scheduled on the node
-
getScheduledCpuForNode
Description copied from interface:ISchedulingStateGet the total cpu currently scheduled on a node.- Specified by:
getScheduledCpuForNodein interfaceISchedulingState- Parameters:
nodeId- the id of the node- Returns:
- the total cpu currently scheduled on the node
-
getINimbus
-
getConf
Description copied from interface:ISchedulingStateGet the nimbus configuration.- Specified by:
getConfin interfaceISchedulingState
-
unassign
Unassign everything for the given topology id.- Parameters:
topoId- the is of the topology to unassign
-
updateFrom
Update the assignments and status from the other cluster.- Parameters:
other- the cluster to get the assignments and status from
-
getMinWorkerCpu
public double getMinWorkerCpu() -
getGreyListedSupervisors
-
setGreyListedSupervisors
-