Hystrix-原理解析

#Hystrix原理浅析

[TOC]

带着问题看原理:

  1. 执行流程
  2. 如何做到信号量隔离和线程隔离
  3. 动态配置的实现
  4. 如何记录信号量数据
  5. 如何实现断路器
  6. 如何实现合并请求

1. Hystrix类结构以及命令执行流程

1. Hystrix类结构

Hystrix的命令实现主体部分是在抽象类AbstractCommnd中,HystrixCommandHystrixObservableCommand是其的实现类。
set up-w600

HystrixCommand:
最终的执行主体是在方法AbstractCommnd.toObservable()中。

1
2
3
4
5
6
7
8
9
10
11
 public R execute() {
try {
return queue().get();
} catch (Exception e) {
throw Exceptions.sneakyThrow(decomposeException(e));
}
}
public Future<R> queue() {
//第一步执行
final Future<R> delegate = **toObservable()**.toBlocking().toFuture();
}

HystrixObservableCommand:

1
2
3
4
5
6
7
8
9
10
public Observable<R> observe() {
ReplaySubject<R> subject = ReplaySubject.create();
final Subscription sourceSubscription = **toObservable()**.subscribe(subject);
return subject.doOnUnsubscribe(new Action0() {
@Override
public void call() {
sourceSubscription.unsubscribe();
}
});
}

2. 执行流程

执行过程分为以下几个步骤:

  1. 首先判断是否开启缓存,如果缓存开启,且可以从缓存中直接查询到结果则直接返回。否则进入下一步;
  2. 第二步 进行断路器判断逻辑:断路器通过健康统计信息来判断是否打开断路器,如果打开则调用降级方法或向外告知调用者,否则继续下一步;
  3. 第三步 隔离控制判断:当请求的数量达到阈值(信号隔离)或者达到线程池上界则拒绝请求调用降级方法或向外告知调用者;并记录此次事件。
  4. 第四步执行业务逻辑,如果此时业务逻辑抛出异常或者超市则会调用降级逻辑或…,并记录此次事件。
  5. 当正常运行结束后也会记录执行成功的事件和运行的时延等信息,一是供断路器使用、二是提供实时监控数据输出

整个执行过程:将每个步骤都放在一个监听器(Action~)里。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
public Observable<R> toObservable() {
final AbstractCommand<R> _cmd = this;
//命令结束的清理工作:发送命令执行结束清理事件
final Action0 terminateCommandCleanup = new Action0() {
@Override
public void call() {
if (_cmd.commandState.compareAndSet(CommandState.OBSERVABLE_CHAIN_CREATED, CommandState.TERMINAL)) {
handleCommandEnd(false); //user code never ran
} else if (_cmd.commandState.compareAndSet(CommandState.USER_CODE_EXECUTED, CommandState.TERMINAL)) {
handleCommandEnd(true); //user code did run
}
}
};
//命令被取消时执行的逻辑。ps:虽然命令取消,但还是会命令还是会被执行完成。只是调用者不再等待
final Action0 unsubscribeCommandCleanup = new Action0() {
@Override
public void call() {
handleCommandEnd(true|false); //user code did run
}
}
};
//执行主体
final Func0<Observable<R>> applyHystrixSemantics = new Func0<Observable<R>>() {
@Override
public Observable<R> call() {
if (commandState.get().equals(CommandState.UNSUBSCRIBED)) {
return Observable.never();
}
//命令执行主体
return applyHystrixSemantics(_cmd);
}
};
//...
return Observable.defer(new Func0<Observable<R>>() {
@Override
public Observable<R> call() {
commandStartTimestamp = System.currentTimeMillis();
//判断缓存是否开启
final boolean requestCacheEnabled = isRequestCachingEnabled();
final String cacheKey = getCacheKey();
//尝试从缓存中获取数据
if (requestCacheEnabled) {
HystrixCommandResponseFromCache<R> fromCache = (HystrixCommandResponseFromCache<R>) requestCache.get(cacheKey);
if (fromCache != null) {
isResponseFromCache = true;
return handleRequestCacheHitAndEmitValues(fromCache, _cmd);
}
}
//执行命令
Observable<R> hystrixObservable =
Observable.defer(applyHystrixSemantics)
.map(wrapWithAllOnNextHooks);

Observable<R> afterCache;
//执行完成后将结果放到缓存中
if (requestCacheEnabled && cacheKey != null) {
// wrap it for caching
HystrixCachedObservable<R> toCache = HystrixCachedObservable.from(hystrixObservable, _cmd);
HystrixCommandResponseFromCache<R> fromCache = (HystrixCommandResponseFromCache<R>) requestCache.putIfAbsent(cacheKey, toCache);
//...
} else {
afterCache = hystrixObservable;
}
//...
}
});
}

执行命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
private Observable<R> applyHystrixSemantics(final AbstractCommand<R> _cmd) {
//断路器判断逻辑
if (circuitBreaker.attemptExecution()) {

final TryableSemaphore executionSemaphore = getExecutionSemaphore();
final AtomicBoolean semaphoreHasBeenReleased = new AtomicBoolean(false);
final Action0 singleSemaphoreRelease = new Action0() {
@Override
public void call() {
if (semaphoreHasBeenReleased.compareAndSet(false, true)) {
executionSemaphore.release();
}
}
};
// 信号量隔离逻辑:
// 当为线程隔离时: 其信号量实现类为 *TryableSemaphoreNoOp* 它会通过所有请求;
// 当为信号量隔离时:实现类为 *TryableSemaphoreActual*,它内部维护AtomicInteger 利用CAS的方式来进行信号量控制
if (executionSemaphore.tryAcquire()) {
try {
//executionResult用来保存执行上下文,比如执行开始时间、结束时间、是否成功等,是记录命令执行情况的依据。
executionResult = executionResult.setInvocationStartTime(System.currentTimeMillis());
return executeCommandAndObserve(_cmd)
.doOnError(markExceptionThrown)
.doOnTerminate(singleSemaphoreRelease)
.doOnUnsubscribe(singleSemaphoreRelease);
} catch (RuntimeException e) {
return Observable.error(e);
}
} else {
//信号量拒绝降级
return handleSemaphoreRejectionViaFallback();
}
} else {
//进行断路器降级
return handleShortCircuitViaFallback();
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
private Observable<R> executeCommandWithSpecifiedIsolation(final AbstractCommand<R> _cmd) {
if (properties.executionIsolationStrategy().get() == ExecutionIsolationStrategy.THREAD) {

return Observable.defer(new Func0<Observable<R>>() {
@Override
public Observable<R> call() {
executionResult = executionResult.setExecutionOccurred();
if (threadState.compareAndSet(ThreadState.NOT_USING_THREAD, ThreadState.STARTED)) {
threadPool.markThreadExecution();
// store the command that is being run
endCurrentThreadExecutingCommand = Hystrix.startCurrentThreadExecutingCommand(getCommandKey());
executionResult = executionResult.setExecutedInThread();

try {
//...
//** 执行用户定义的业务代码,对于HystrixCommand和HystrixObservableCommand实现不同
return getUserExecutionObservable(_cmd);
} catch (Throwable ex) {
return Observable.error(ex);
}
}...
}
// RxJava帮我们做了线程切换,此处就是线程切换的地方
}).**subscribeOn**(threadPool.getScheduler(new Func0<Boolean>() {
@Override
public Boolean call() {
return properties.executionIsolationThreadInterruptOnTimeout().get() && _cmd.isCommandTimedOut.get() == TimedOutStatus.TIMED_OUT;
}
}));

// HystrixObservableCommand默认是信号量隔离
} else {
return Observable.defer(new Func0<Observable<R>>() {
@Override
public Observable<R> call() {
executionResult = executionResult.setExecutionOccurred();
//...
try {
executionHook.onRunStart(_cmd);
executionHook.onExecutionStart(_cmd);
return getUserExecutionObservable(_cmd);
}...
}
});
}
}

AbstractCommand中有四大关键功能属性:

1
2
3
4
5
6
7
8
//配置
HystrixCommandProperties properties;
//断路器
HystrixCircuitBreaker circuitBreaker;
//命令服务指标计量器
HystrixCommandMetrics metrics;
//线程池
HystrixThreadPool threadPool;

怎样实现动态配置?

先来看怎么动态配置属性(这是Netflix形成的一套体系,不用注解、反射 ):
看Hystrix中线程池的🌰:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
class HystrixThreadPoolDefault implements HystrixThreadPool {
private static final Logger logger = LoggerFactory.getLogger(HystrixThreadPoolDefault.class);

private final HystrixThreadPoolProperties properties;
private final BlockingQueue<Runnable> queue;
private final ThreadPoolExecutor threadPool;
private final HystrixThreadPoolMetrics metrics;
private final int queueSize;

private void touchConfig() {
//拉取最新配置信息
final int dynamicCoreSize = properties.coreSize().get();
final int configuredMaximumSize = properties.maximumSize().get();
int dynamicMaximumSize = properties.actualMaximumSize();
final boolean allowSizesToDiverge = properties.getAllowMaximumSizeToDivergeFromCoreSize().get();
boolean maxTooLow = false;

if (allowSizesToDiverge && configuredMaximumSize < dynamicCoreSize) {
dynamicMaximumSize = dynamicCoreSize;
maxTooLow = true;
}

if (threadPool.getCorePoolSize() != dynamicCoreSize || (allowSizesToDiverge && threadPool.getMaximumPoolSize() != dynamicMaximumSize)) {
//设置属性
threadPool.setCorePoolSize(dynamicCoreSize);
threadPool.setMaximumPoolSize(dynamicMaximumSize);
}
threadPool.setKeepAliveTime(properties.keepAliveTimeMinutes().get(), TimeUnit.MINUTES);
}
//每次使用线程池时,会去重新加载下线程池的配置,并对线程池属性进行修改
@Override
public ThreadPoolExecutor getExecutor() {
touchConfig();
return threadPool;
}

每次获取线程池之前会刷新一次线程池的配置

HystrixThreadPoolProperties

1
2
3
4
5
6
7
8
HystrixProperty<Integer> corePoolSize;
HystrixProperty<Integer> maximumPoolSize;
HystrixProperty<Integer> keepAliveTime;
HystrixProperty<Integer> maxQueueSize;
HystrixProperty<Integer> queueSizeRejectionThreshold;
HystrixProperty<Boolean> allowMaximumSizeToDivergeFromCoreSize;
HystrixProperty<Integer> threadPoolRollingNumberStatisticalWindowInMilliseconds;
HystrixProperty<Integer> threadPoolRollingNumberStatisticalWindowBuckets;

Hystrix 默认使用 Archaius来实现属性的配置:
HystrixProperty<R>的实现是ArchaiusDynamicProperty的子类(StringDynamicPropertyIntegerDynamicPropertyLongDynamicPropertyBooleanDynamicProperty)。
ArchaiusDynamicProperty有个静态容器:final static ConcurrentHashMap<String, DynamicProperty> ALL_PROPS 其用来包含所有动态属性。

  • 属性获取
    当通过HystrixProperty<R>.getValue(name)获取配置值时,会先去ALL_PROPS中寻找动态配置的值,如果没有设置则会使用ArchaiusDynamicProperty中初始化时设置的默认值。
  • 属性修改
    当我们使用ConfigurationManager去修改配置值时,会通过nameALL_PROPS中找到DynamicProperty并设置值即可生效。

HystrixCircuitBreaker:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
@Override
public boolean attemptExecution() {
if (properties.circuitBreakerForceOpen().get()) {
return false;
}
if (properties.circuitBreakerForceClosed().get()) {
return true;
}
//此处是关键 断路器打开的时间。
if (circuitOpened.get() == -1) {
return true;
} else {
//若休眠时间已过,则会放过一次请求,如果放过的请求没成功则会继续封禁,且会调整断路器打开时间为当前时间,延长断路器时间。
if (isAfterSleepWindow()) {
if (status.compareAndSet(Status.OPEN, Status.HALF_OPEN)) {
return true;
} else {
return false;
}
} else {
return false;
}
}
}
private boolean isAfterSleepWindow() {
final long circuitOpenTime = circuitOpened.get();
final long currentTime = System.currentTimeMillis();
//如果当前时间已经大于休眠时间
final long sleepWindowTime = properties.circuitBreakerSleepWindowInMilliseconds().get();
return currentTime > circuitOpenTime + sleepWindowTime;
}

private Subscription subscribeToStream() {
//订阅Metrics中的**HealthCountsStream**数据流产出的HealthCounts数据
return metrics.getHealthCountsStream()
.observe()
.subscribe(new Subscriber<HealthCounts>() {
@Override
public void onNext(HealthCounts hc) {
//当总请求量到达一定程度的时候才会进行后续的判断:冷启动
if (hc.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
} else {
//当错误率到达一定程度的时候,将断路器设置为打开状态
if (hc.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
} else {
if (status.compareAndSet(Status.CLOSED, Status.OPEN)) {
circuitOpened.set(System.currentTimeMillis());
}
}
}
}
});
}
class HealthCounts {
private final long totalCount;
private final long errorCount;
private final int errorPercentage;
}

HystrixCommandMetrics

HystrixCommandMetrics是Hystrix中比较精彩的一部分:
它定义了如下几个事件流:

1
2
3
4
5
6
7
8
9
10
11
12
//健康数量流(滑动窗口)
HealthCountsStream healthCountsStream;
//事件数量流(滑动窗口)
RollingCommandEventCounterStream rollingCommandEventCounterStream;
//事件数量流(从VM启动开始的数据)
CumulativeCommandEventCounterStream cumulativeCommandEventCounterStream;
//命令执行延时分布流(滑动窗口)
RollingCommandLatencyDistributionStream rollingCommandLatencyDistributionStream;
//命令执行延时分布流(滑动窗口)
RollingCommandUserLatencyDistributionStream rollingCommandUserLatencyDistributionStream;
//最大并发流(滑动窗口)
RollingCommandMaxConcurrencyStream rollingCommandMaxConcurrencyStream;

下面,本文将以HealthCountsStream为例来介绍它是怎样获取命令执行信息,并通过滑动窗口的形式聚合信息成供断路器使用的HealthCounts

  • 获取命令执行信息
    在每个命令执行完成(成功或失败)后,都会执行handleCommandEnd方法。此方法会记录命令执行时长信息到执行结果ExecutionResult中,并把ExecutionResult告诉Metrics

Metrics会调用HystrixThreadEventStream.executionDone()方法创建一个命令执行完成对象HystrixCommandCompletion,并把这个对象写入事件流HystrixCommandCompletionStream中(对于每个命令,Hystrix都会保存此命令的唯一HystrixCommandCompletionStream用于接受和发送命令完成事件的流)。

1
2
3
4
5
6
7
8
9
10
11
12
13
private void handleCommandEnd(boolean commandExecutionStarted) {
//记录命令执行时间
long userThreadLatency = System.currentTimeMillis() - commandStartTimestamp;
// executionResult:为命令的执行结果:包含命令状态信息(成功、失败、超时...)
executionResult = executionResult.markUserThreadCompletion((int) userThreadLatency);
//告诉Metrics记录命令执行完成
metrics.markCommandDone(executionResult, commandKey, threadPoolKey, commandExecutionStarted);
//...
}

HystrixCommandCompletion event = HystrixCommandCompletion.from(executionResult, commandKey, threadPoolKey);
HystrixCommandCompletionStream commandStream = HystrixCommandCompletionStream.getInstance(commandKey);
commandStream.write(event);

此时,所有需要获取命令执行信息,只需要成为HystrixCommandCompletionStream的订阅者即可。

HystrixCommandCompletionStream中通过SerializedSubject来保证多线程并发写不会引起乱序等问题。new SerializedSubject<HystrixCommandCompletion, HystrixCommandCompletion>(PublishSubject.<HystrixCommandCompletion>create())

  • 消费命令信息
    下面以HealthCountsStream为例来介绍其是怎么消费命令执行信息,并输出可供断路器使用的信息的:
1
2
3
4
5
6
7
8
9
10
11
12
private HealthCountsStream(final HystrixCommandKey commandKey, final int numBuckets, final int bucketSizeInMs,
Func2<long[], HystrixCommandCompletion, long[]> reduceCommandCompletion) {
//获取某个命令的`HystrixCommandCompletionStream`作为数据源
super(HystrixCommandCompletionStream.getInstance(commandKey), numBuckets, bucketSizeInMs, reduceCommandCompletion, healthCheckAccumulator);
}

private static final Func2<HealthCounts, long[], HealthCounts> healthCheckAccumulator = new Func2<HealthCounts, long[], HealthCounts>() {
@Override
public HealthCounts call(HealthCounts healthCounts, long[] bucketEventCounts) {
return healthCounts.plus(bucketEventCounts);
}
};

set up-w500
跟进其父类BucketedCounterStreamBucketedRollingCounterStream,可以看到命令完成事件数据流是怎么被消费的。

首先,BucketedCounterStream是用来按指定的时间片段长度来生成Bucketed数据流(保留一个时间片段内的所有执行结果类型命令请求数)。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
protected BucketedCounterStream(final HystrixEventStream<Event> inputEventStream, final int numBuckets, final int bucketSizeInMs,
final Func2<Bucket, Event, Bucket> appendRawEventToBucket) {
this.numBuckets = numBuckets;
this.reduceBucketToSummary = new Func1<Observable<Event>, Observable<Bucket>>() {
@Override
public Observable<Bucket> call(Observable<Event> eventBucket) {
//叠加操作
return eventBucket.reduce(getEmptyBucketSummary(), appendRawEventToBucket);
}
};
this.bucketedStream = Observable.defer(new Func0<Observable<Bucket>>() {
@Override
public Observable<Bucket> call() {
return hystrixCommandCompletionStream
.observe()
//关键!RxJava.window 以bucketSizeInMs时间大小为一个窗口发射数据
.window(bucketSizeInMs, TimeUnit.MILLISECONDS)
// 将所有的命令请求聚合成一个总的结果:(每个命令执行结果类型的数量)
.flatMap(reduceBucketToSummary)
.startWith(emptyEventCountsToStart);
}
});
}

BucketedRollingCounterStream 会订阅BucketedCounterStream生成的bucketedStream流,并基于此做了滑动窗口的功能,并聚合欢动窗口内的所有命令执行结果类型计量信息。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
protected BucketedRollingCounterStream(HystrixEventStream<Event> stream, final int numBuckets, int bucketSizeInMs,
final Func2<Bucket, Event, Bucket> appendRawEventToBucket,
final Func2<Output, Bucket, Output> reduceBucket) {
//此处 appendRawEventToBucket是实现`HealthCounts`相加的函数
super(stream, numBuckets, bucketSizeInMs, appendRawEventToBucket);
Func1<Observable<Bucket>, Observable<Output>> reduceWindowToSummary = new Func1<Observable<Bucket>, Observable<Output>>() {
@Override
public Observable<Output> call(Observable<Bucket> window) {
return window.scan(getEmptyOutputValue(), reduceBucket).skip(numBuckets);
}
};
this.sourceStream = bucketedStream
//滑动窗口功能
.window(numBuckets, 1)
.flatMap(reduceWindowToSummary)//...
}

再回过头来看断路器逻辑中,它监听的即是BucketedRollingCounterStream中的sourceStream。它向外推送appendRawEventToBucket的结果数据。

HystrixCollapser:

Collapser的原理是:将一个时间窗内的所有请求合并批量发送到依赖方进行请求返回。
set up-w500
HystrixCollapser有两种域范围:用户请求范围、全局范围。用户请求范围只会把用户请求范围内的请求合并,全局范围则会合并所有请求
整个过程的时序图如下:

1
2
3
4
5
6
HystrixCollapser->RequestCollapser: toObservable
RequestCollapser->HystrixTimer: submitRequest
Note right of HystrixTimer: 第一次提交请求时,会创建HystrixTimer,并开启定时任务(**addTimerListener**)
HystrixCollapser->RequestBatch: offer,将请求加到RequestBatch队列中(如果队列中已经有了此请求且设置了请求缓存,则直接返回之前的;当队列的大小大于批量的上界时执行任务,并重新开启下一个批量),执行线程阻塞。
HystrixCollapser->HystrixCollapser: createNewBatchAndExecutePreviousIfNeeded,当时间窗口时间到达,或者到达最大请求时,则执行批量请求
HystrixCollapser->RequestBatch: executeBatchIfNotAlreadyStarted,创建批量请求命令,提交请求。执行完成后并将结果映射会原有请求

HystrixCommandProperties命令列表:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
HystrixProperty<Integer> circuitBreakerRequestVolumeThreshold; // number of requests that must be made within a statisticalWindow before open/close decisions are made using stats
HystrixProperty<Integer> circuitBreakerSleepWindowInMilliseconds; // milliseconds after tripping circuit before allowing retry
HystrixProperty<Boolean> circuitBreakerEnabled; // Whether circuit breaker should be enabled.
HystrixProperty<Integer> circuitBreakerErrorThresholdPercentage; // % of 'marks' that must be failed to trip the circuit
HystrixProperty<Boolean> circuitBreakerForceOpen; // a property to allow forcing the circuit open (stopping all requests)
HystrixProperty<Boolean> circuitBreakerForceClosed; // a property to allow ignoring errors and therefore never trip 'open' (ie. allow all traffic through)
HystrixProperty<ExecutionIsolationStrategy> executionIsolationStrategy; // Whether a command should be executed in a separate thread or not.
HystrixProperty<Integer> executionTimeoutInMilliseconds; // Timeout value in milliseconds for a command
HystrixProperty<Boolean> executionTimeoutEnabled; //Whether timeout should be triggered
HystrixProperty<String> executionIsolationThreadPoolKeyOverride; // What thread-pool this command should run in (if running on a separate thread).
HystrixProperty<Integer> executionIsolationSemaphoreMaxConcurrentRequests; // Number of permits for execution semaphore
HystrixProperty<Integer> fallbackIsolationSemaphoreMaxConcurrentRequests; // Number of permits for fallback semaphore
HystrixProperty<Boolean> fallbackEnabled; // Whether fallback should be attempted.
HystrixProperty<Boolean> executionIsolationThreadInterruptOnTimeout; // Whether an underlying Future/Thread (when runInSeparateThread == true) should be interrupted after a timeout
HystrixProperty<Boolean> executionIsolationThreadInterruptOnFutureCancel; // Whether canceling an underlying Future/Thread (when runInSeparateThread == true) should interrupt the execution thread
HystrixProperty<Integer> metricsRollingStatisticalWindowInMilliseconds; // milliseconds back that will be tracked
HystrixProperty<Integer> metricsRollingStatisticalWindowBuckets; // number of buckets in the statisticalWindow
HystrixProperty<Boolean> metricsRollingPercentileEnabled; // Whether monitoring should be enabled (SLA and Tracers).
HystrixProperty<Integer> metricsRollingPercentileWindowInMilliseconds; // number of milliseconds that will be tracked in RollingPercentile
HystrixProperty<Integer> metricsRollingPercentileWindowBuckets; // number of buckets percentileWindow will be divided into
HystrixProperty<Integer> metricsRollingPercentileBucketSize; // how many values will be stored in each percentileWindowBucket
HystrixProperty<Integer> metricsHealthSnapshotIntervalInMilliseconds; // time between health snapshots
HystrixProperty<Boolean> requestLogEnabled; // whether command request logging is enabled.
HystrixProperty<Boolean> requestCacheEnabled; // Whether request caching is enabled.
您的支持是我创作源源不断的动力