public class StormClusterStateImpl extends Object implements IStormClusterState
| Constructor and Description |
|---|
StormClusterStateImpl(IStateStorage stateStorage,
ILocalAssignmentsBackend assignmentsassignmentsBackend,
ClusterStateContext context,
boolean shouldCloseStateStorageOnDisconnect) |
| Modifier and Type | Method and Description |
|---|---|
void |
activateStorm(String stormId,
StormBase stormBase,
Map<String,Object> topoConf) |
List<String> |
activeKeys() |
List<String> |
activeStorms() |
void |
addNimbusHost(String nimbusId,
NimbusSummary nimbusSummary) |
void |
addPrivateWorkerKey(WorkerTokenServiceType type,
String topologyId,
long keyVersion,
PrivateWorkerKey key)
Store a new version of a private key.
|
Assignment |
assignmentInfo(String stormId,
Runnable callback)
Get the assignment based on storm id from local backend.
|
VersionedData<Assignment> |
assignmentInfoWithVersion(String stormId,
Runnable callback) |
List<String> |
assignments(Runnable callback) |
Map<String,Assignment> |
assignmentsInfo()
Get all the topologies assignments mapping stormId -> Assignment from local backend.
|
Integer |
assignmentVersion(String stormId,
Runnable callback) |
List<String> |
backpressureTopologies()
Get backpressure topologies.
|
List<String> |
blobstore(Runnable callback) |
List<String> |
blobstoreInfo(String blobKey) |
Credentials |
credentials(String stormId,
Runnable callback) |
void |
deleteTopologyProfileRequests(String stormId,
ProfileRequest profileRequest) |
void |
disconnect() |
List<ErrorInfo> |
errors(String stormId,
String componentId) |
List<String> |
errorTopologies() |
Map<ExecutorInfo,ExecutorBeat> |
executorBeats(String stormId,
Map<List<Long>,NodeInfo> executorNodePort)
need to take executor->node+port in explicitly so that we don’t run into a situation where a long dead worker with a skewed clock overrides all the timestamps.
|
NimbusInfo |
getLeader(Runnable callback)
Get leader info from state store, which was written when a master gains leadership.
|
long |
getNextPrivateWorkerKeyVersion(WorkerTokenServiceType type,
String topologyId)
Get the next key version number that should be used for this topology id.
|
PrivateWorkerKey |
getPrivateWorkerKey(WorkerTokenServiceType type,
String topologyId,
long keyVersion)
Get a private key used to validate a token is correct.
|
List<ProfileRequest> |
getTopologyProfileRequests(String stormId) |
ClusterWorkerHeartbeat |
getWorkerHeartbeat(String stormId,
String node,
Long port) |
List<ProfileRequest> |
getWorkerProfileRequests(String stormId,
NodeInfo nodeInfo) |
List<String> |
heartbeatStorms() |
Set<String> |
idsOfTopologiesWithPrivateWorkerKeys()
Get a list of all topologyIds that currently have private worker keys stored, of any kind.
|
boolean |
isAssignmentsBackendSynchronized()
Flag to indicate if the assignments synced successfully, see
IStormClusterState.syncRemoteAssignments(Map). |
boolean |
isPacemakerStateStore()
Flag to indicate if the Pacameker is backend store.
|
protected void |
issueCallback(AtomicReference<Runnable> cb) |
protected void |
issueMapCallback(ConcurrentHashMap<String,Runnable> callbackConcurrentHashMap,
String key) |
ErrorInfo |
lastError(String stormId,
String componentId) |
List<NimbusSummary> |
nimbuses() |
Assignment |
remoteAssignmentInfo(String stormId,
Runnable callback)
Get the assignment based on storm id from remote state store, eg: ZK.
|
void |
removeAllPrivateWorkerKeys(String topologyId)
Remove all of the worker keys for a given topology.
|
void |
removeBackpressure(String stormId)
Remove backpressure.
|
void |
removeBlobstoreKey(String blobKey) |
void |
removeExpiredPrivateWorkerKeys(String topologyId)
Remove all keys for the given topology that have expired.
|
void |
removeKeyVersion(String blobKey) |
void |
removeStorm(String stormId) |
void |
removeStormBase(String stormId) |
void |
removeWorkerBackpressure(String stormId,
String node,
Long port)
Remove worker backpressure.
|
void |
removeWorkerHeartbeat(String stormId,
String node,
Long port) |
void |
reportError(String stormId,
String componentId,
String node,
Long port,
Throwable error) |
void |
setAssignment(String stormId,
Assignment info,
Map<String,Object> topoConf) |
void |
setAssignmentsBackendSynchronized()
Mark the assignments as synced successfully, see
IStormClusterState.isAssignmentsBackendSynchronized(). |
void |
setCredentials(String stormId,
Credentials creds,
Map<String,Object> topoConf) |
void |
setTopologyLogConfig(String stormId,
LogConfig logConfig,
Map<String,Object> topoConf) |
void |
setupBackpressure(String stormId,
Map<String,Object> topoConf)
Setup backpressure.
|
void |
setupBlob(String key,
NimbusInfo nimbusInfo,
Integer versionInfo) |
void |
setupErrors(String stormId,
Map<String,Object> topoConf) |
void |
setupHeatbeats(String stormId,
Map<String,Object> topoConf) |
void |
setWorkerProfileRequest(String stormId,
ProfileRequest profileRequest) |
StormBase |
stormBase(String stormId,
Runnable callback)
Get a storm base for a topology.
|
String |
stormId(String stormName)
Get storm id from passed name, null if the name doesn’t exist on cluster.
|
void |
supervisorHeartbeat(String supervisorId,
SupervisorInfo info) |
SupervisorInfo |
supervisorInfo(String supervisorId) |
List<String> |
supervisors(Runnable callback) |
void |
syncRemoteAssignments(Map<String,byte[]> remote)
Sync the remote state store assignments to local backend, used when master gains leadership, see
LeaderListenerCallback. |
void |
syncRemoteIds(Map<String,String> remote)
Sync all the active storm ids of the cluster, used now when master gains leadership.
|
void |
teardownHeartbeats(String stormId) |
void |
teardownTopologyErrors(String stormId) |
boolean |
topologyBackpressure(String stormId,
long timeoutMs,
Runnable callback)
Check whether a topology is in throttle-on status or not: if the backpresure/storm-id dir is not empty, this topology has throttle-on, otherwise throttle-off.
|
LogConfig |
topologyLogConfig(String stormId,
Runnable cb) |
void |
updateStorm(String stormId,
StormBase newElems)
To update this function due to APersistentMap/APersistentSet is clojure’s structure.
|
void |
workerHeartbeat(String stormId,
String node,
Long port,
ClusterWorkerHeartbeat info) |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitallSupervisorInfo, allSupervisorInfo, getTopoId, topologyBasespublic StormClusterStateImpl(IStateStorage stateStorage, ILocalAssignmentsBackend assignmentsassignmentsBackend, ClusterStateContext context, boolean shouldCloseStateStorageOnDisconnect) throws Exception
Exceptionprotected void issueCallback(AtomicReference<Runnable> cb)
protected void issueMapCallback(ConcurrentHashMap<String,Runnable> callbackConcurrentHashMap, String key)
public List<String> assignments(Runnable callback)
assignments in interface IStormClusterStatepublic Assignment assignmentInfo(String stormId, Runnable callback)
IStormClusterStateGet the assignment based on storm id from local backend.
assignmentInfo in interface IStormClusterStatestormId - topology idcallback - callback functionAssignmentpublic Assignment remoteAssignmentInfo(String stormId, Runnable callback)
IStormClusterStateGet the assignment based on storm id from remote state store, eg: ZK.
remoteAssignmentInfo in interface IStormClusterStatestormId - topology idcallback - callback functionAssignmentpublic Map<String,Assignment> assignmentsInfo()
IStormClusterStateGet all the topologies assignments mapping stormId -> Assignment from local backend.
assignmentsInfo in interface IStormClusterStatepublic void syncRemoteAssignments(Map<String,byte[]> remote)
IStormClusterStateSync the remote state store assignments to local backend, used when master gains leadership, see LeaderListenerCallback.
syncRemoteAssignments in interface IStormClusterStateremote - assigned assignments for a specific IStormClusterState instance, usually a supervisor/node.public boolean isAssignmentsBackendSynchronized()
IStormClusterStateFlag to indicate if the assignments synced successfully, see IStormClusterState.syncRemoteAssignments(Map).
isAssignmentsBackendSynchronized in interface IStormClusterStatepublic boolean isPacemakerStateStore()
IStormClusterStateFlag to indicate if the Pacameker is backend store.
isPacemakerStateStore in interface IStormClusterStatepublic void setAssignmentsBackendSynchronized()
IStormClusterStateMark the assignments as synced successfully, see IStormClusterState.isAssignmentsBackendSynchronized().
setAssignmentsBackendSynchronized in interface IStormClusterStatepublic VersionedData<Assignment> assignmentInfoWithVersion(String stormId, Runnable callback)
assignmentInfoWithVersion in interface IStormClusterStatepublic Integer assignmentVersion(String stormId, Runnable callback) throws Exception
assignmentVersion in interface IStormClusterStateExceptionpublic List<String> blobstoreInfo(String blobKey)
blobstoreInfo in interface IStormClusterStatepublic List<NimbusSummary> nimbuses()
nimbuses in interface IStormClusterStatepublic void addNimbusHost(String nimbusId, NimbusSummary nimbusSummary)
addNimbusHost in interface IStormClusterStatepublic List<String> activeStorms()
activeStorms in interface IStormClusterStatepublic StormBase stormBase(String stormId, Runnable callback)
IStormClusterStateGet a storm base for a topology.
stormBase in interface IStormClusterStatestormId - the id of the topologycallback - something to call if the data changes (best effort)public String stormId(String stormName)
IStormClusterStateGet storm id from passed name, null if the name doesn’t exist on cluster.
stormId in interface IStormClusterStatestormName - storm namepublic void syncRemoteIds(Map<String,String> remote)
IStormClusterStateSync all the active storm ids of the cluster, used now when master gains leadership.
syncRemoteIds in interface IStormClusterStateremote - stormName -> stormId mappingpublic ClusterWorkerHeartbeat getWorkerHeartbeat(String stormId, String node, Long port)
getWorkerHeartbeat in interface IStormClusterStatepublic List<ProfileRequest> getWorkerProfileRequests(String stormId, NodeInfo nodeInfo)
getWorkerProfileRequests in interface IStormClusterStatepublic List<ProfileRequest> getTopologyProfileRequests(String stormId)
getTopologyProfileRequests in interface IStormClusterStatepublic void setWorkerProfileRequest(String stormId, ProfileRequest profileRequest)
setWorkerProfileRequest in interface IStormClusterStatepublic void deleteTopologyProfileRequests(String stormId, ProfileRequest profileRequest)
deleteTopologyProfileRequests in interface IStormClusterStatepublic Map<ExecutorInfo,ExecutorBeat> executorBeats(String stormId, Map<List<Long>,NodeInfo> executorNodePort)
need to take executor->node+port in explicitly so that we don’t run into a situation where a long dead worker with a skewed clock overrides all the timestamps. By only checking heartbeats with an assigned node+port, and only reading executors from that heartbeat that are actually assigned, we avoid situations like that.
executorBeats in interface IStormClusterStatestormId - topology idexecutorNodePort - executor id -> node + portpublic List<String> supervisors(Runnable callback)
supervisors in interface IStormClusterStatepublic SupervisorInfo supervisorInfo(String supervisorId)
supervisorInfo in interface IStormClusterStatepublic void setupHeatbeats(String stormId, Map<String,Object> topoConf)
setupHeatbeats in interface IStormClusterStatepublic void teardownHeartbeats(String stormId)
teardownHeartbeats in interface IStormClusterStatepublic void teardownTopologyErrors(String stormId)
teardownTopologyErrors in interface IStormClusterStatepublic NimbusInfo getLeader(Runnable callback)
IStormClusterStateGet leader info from state store, which was written when a master gains leadership.
Caution: it can not be used for fencing and is only for informational purposes because we use ZK as our backend now, which could have a overdue info of nodes.
getLeader in interface IStormClusterStatecallback - callback funcNimbusInfopublic List<String> backpressureTopologies()
IStormClusterStateGet backpressure topologies.
backpressureTopologies in interface IStormClusterStatepublic List<String> heartbeatStorms()
heartbeatStorms in interface IStormClusterStatepublic List<String> errorTopologies()
errorTopologies in interface IStormClusterStatepublic void setTopologyLogConfig(String stormId, LogConfig logConfig, Map<String,Object> topoConf)
setTopologyLogConfig in interface IStormClusterStatepublic LogConfig topologyLogConfig(String stormId, Runnable cb)
topologyLogConfig in interface IStormClusterStatepublic void workerHeartbeat(String stormId, String node, Long port, ClusterWorkerHeartbeat info)
workerHeartbeat in interface IStormClusterStatepublic void removeWorkerHeartbeat(String stormId, String node, Long port)
removeWorkerHeartbeat in interface IStormClusterStatepublic void supervisorHeartbeat(String supervisorId, SupervisorInfo info)
supervisorHeartbeat in interface IStormClusterStatepublic boolean topologyBackpressure(String stormId, long timeoutMs, Runnable callback)
Check whether a topology is in throttle-on status or not: if the backpresure/storm-id dir is not empty, this topology has throttle-on, otherwise throttle-off. But if the backpresure/storm-id dir is not empty and has not been updated for more than timeoutMs, we treat it as throttle-off. This will prevent the spouts from getting stuck indefinitely if something wrong happens.
topologyBackpressure in interface IStormClusterStatestormId - The topology IdtimeoutMs - How long until the backpressure znode is invalid.callback - The callback functionpublic void setupBackpressure(String stormId, Map<String,Object> topoConf)
IStormClusterStateSetup backpressure.
setupBackpressure in interface IStormClusterStatepublic void removeBackpressure(String stormId)
IStormClusterStateRemove backpressure.
removeBackpressure in interface IStormClusterStatepublic void removeWorkerBackpressure(String stormId, String node, Long port)
IStormClusterStateRemove worker backpressure.
removeWorkerBackpressure in interface IStormClusterStatepublic void activateStorm(String stormId, StormBase stormBase, Map<String,Object> topoConf)
activateStorm in interface IStormClusterStatepublic void updateStorm(String stormId, StormBase newElems)
To update this function due to APersistentMap/APersistentSet is clojure’s structure.
updateStorm in interface IStormClusterStatepublic void removeStormBase(String stormId)
removeStormBase in interface IStormClusterStatepublic void setAssignment(String stormId, Assignment info, Map<String,Object> topoConf)
setAssignment in interface IStormClusterStatepublic void setupBlob(String key, NimbusInfo nimbusInfo, Integer versionInfo)
setupBlob in interface IStormClusterStatepublic List<String> activeKeys()
activeKeys in interface IStormClusterStatepublic List<String> blobstore(Runnable callback)
blobstore in interface IStormClusterStatepublic void removeStorm(String stormId)
removeStorm in interface IStormClusterStatepublic void removeBlobstoreKey(String blobKey)
removeBlobstoreKey in interface IStormClusterStatepublic void removeKeyVersion(String blobKey)
removeKeyVersion in interface IStormClusterStatepublic void setupErrors(String stormId, Map<String,Object> topoConf)
setupErrors in interface IStormClusterStatepublic void reportError(String stormId, String componentId, String node, Long port, Throwable error)
reportError in interface IStormClusterStatepublic List<ErrorInfo> errors(String stormId, String componentId)
errors in interface IStormClusterStatepublic ErrorInfo lastError(String stormId, String componentId)
lastError in interface IStormClusterStatepublic void setCredentials(String stormId, Credentials creds, Map<String,Object> topoConf)
setCredentials in interface IStormClusterStatepublic Credentials credentials(String stormId, Runnable callback)
credentials in interface IStormClusterStatepublic void disconnect()
disconnect in interface IStormClusterStatepublic PrivateWorkerKey getPrivateWorkerKey(WorkerTokenServiceType type, String topologyId, long keyVersion)
IStormClusterStateGet a private key used to validate a token is correct. This is expected to be called from a privileged daemon, and the ACLs should be set up to only allow nimbus and these privileged daemons access to these private keys.
getPrivateWorkerKey in interface IStormClusterStatetype - the type of service the key is for.topologyId - the topology id the key is for.keyVersion - the version of the key this is for.public void addPrivateWorkerKey(WorkerTokenServiceType type, String topologyId, long keyVersion, PrivateWorkerKey key)
IStormClusterStateStore a new version of a private key. This is expected to only ever be called from nimbus. All ACLs however need to be setup to allow the given services access to the stored information.
addPrivateWorkerKey in interface IStormClusterStatetype - the type of service this key is for.topologyId - the topology this key is forkeyVersion - the version of the key this is for.key - the key to store.public long getNextPrivateWorkerKeyVersion(WorkerTokenServiceType type, String topologyId)
IStormClusterStateGet the next key version number that should be used for this topology id. This is expected to only ever be called from nimbus, but it is acceptable if the ACLs are setup so that it can work from a privileged daemon for the given service.
getNextPrivateWorkerKeyVersion in interface IStormClusterStatetype - the type of service this is for.topologyId - the topology id this is for.public void removeExpiredPrivateWorkerKeys(String topologyId)
IStormClusterStateRemove all keys for the given topology that have expired. The number of keys should be small enough that doing an exhaustive scan of them all is acceptable as there is no guarantee that expiration time and version number are related. This should be for all service types. This is expected to only ever be called from nimbus and some ACLs may be setup so being called from other daemons will cause it to fail.
removeExpiredPrivateWorkerKeys in interface IStormClusterStatetopologyId - the id of the topology to scan.public void removeAllPrivateWorkerKeys(String topologyId)
IStormClusterStateRemove all of the worker keys for a given topology. Used to clean up after a topology finishes. This is expected to only ever be called from nimbus and ideally should only ever work from nimbus.
removeAllPrivateWorkerKeys in interface IStormClusterStatetopologyId - the topology to clean up after.public Set<String> idsOfTopologiesWithPrivateWorkerKeys()
IStormClusterStateGet a list of all topologyIds that currently have private worker keys stored, of any kind. This is expected to only ever be called from nimbus.
idsOfTopologiesWithPrivateWorkerKeys in interface IStormClusterStateCopyright © 2021 The Apache Software Foundation. All rights reserved.