Spark內核設計的藝術：架構設計與實現 ( 簡體 字) |
| 作者：耿嘉安 著 | 類別：1. -> 程式設計 -> Spark |
| 譯者： |
| 出版社：機械工業出版社 |  3dWoo書號： 48182
詢問書籍請說出此書號！【缺書】
NT售價： 695 元 |
| 出版日：12/1/2017 |
| 頁數：690 |
| 光碟數：0 |
|
站長推薦：  |
| 印刷：黑白印刷 | 語系： ( 簡體 版 ) |
| |
加入購物車  │加到我的最愛
(請先登入會員) |
| ISBN：9787111584391 |
| 作者序 | 譯者序 | 前言 | 內容簡介 | 目錄 | 序 |
| (簡體書上所述之下載連結耗時費功, 恕不適用在台灣, 若讀者需要請自行嘗試, 恕不保證) |
| 作者序： |
| 譯者序： |
前言：為什么寫這本書
給本書寫前言時，讓我想起了兩年前給《深入理解Spark：核心思想與源碼分析》一書寫前言的經歷。我不禁想起崔護的《題都城南莊》這首詩，詩的內容是：
去年今日此門中，人面桃花相映紅。
人面不知何處去，桃花依舊笑春風。
從核心思想和架構來看，Spark依然是那個Spark，但是我已經找了一個新的“東家”。我的年齡不知不覺中又長了兩歲，Spark也在大數據領域從“新貴”變成了“老人”。Spark的版本從0.x.x到2.x.x基本上也是用了兩年時間。
自從《深入理解Spark：核心思想與源碼分析》一書出版后，引起了一些市場反響，更難得的是得到了很多讀者的反饋。一些熱心的讀者通過微信或者郵件向我指出了書中內容的很多不足之處，包括錯別字、錯誤的描述、代碼分析有點像流水賬、提綱挈領的內容偏少、代碼版本過低等。一些錯誤在修訂的版本中得到了解決，有些修正的內容則通過單獨寫博客來補充。在與讀者的溝通過程中，也糾正了我對一些問題的理解偏差。再次深深地感謝廣大讀者的支持與幫助！
一些讀者對《深入理解Spark：核心思想與源碼分析》一書的內容非常肯定，希望能夠出第2版，高婧雅編輯也一再“慫恿”我，但是我一直沒有寫第2版的打算。我當時希望有人能夠以更好的方式寫一本介紹和分析Spark 2.0版本的源碼分析書籍，因為我感覺之前的寫作方式的確不是很好。在我心中一直有個矛盾：如果源碼太少，源碼分析的書籍將退化成單純講原理的書籍，對于想深入理解Spark實現的讀者來說這是不夠的；如果源碼太多，又讓人有堆砌代碼或者“混”篇幅的感覺。很多源碼分析的書只是簡單說說接口或者方法的功能，讓人始終有種“霧里看花”的感覺。所以我一直很期待能有更好的方式來寫作源碼分析類的書。
在一年多的等待中，我始終沒有發現類似書籍的出現，于是我打算再做一次嘗試。這次擯棄了《深入理解Spark：核心思想與源碼分析》一書中按照代碼執行流程分析的方式，改為先從整體上介紹一個系統，然后逐個分析每個組件的功能，最后將這些組件之間的關系用流程圖的方式串聯起來。本書的寫作方式依然犯有代碼過多的“毛病”，但我還是期待本書能帶來一些新的氣象。
本書的主要特色
按照源碼分析的習慣設計，從腳本分析到初始化，再到核心內容。整個過程遵循由淺入深的基本思路。
每一章先對本章的內容有個總體介紹，然后深入分析各個組件的實現原理，最后將各個組件之間的關系通過執行流程來展現。
本書盡可能地用圖來展示原理，以加速讀者對內容的掌握。
本書講解的很多實現及原理都值得借鑒，可以幫助讀者提升架構設計、程序設計等方面的能力。
本書盡可能保留較多的源碼，以便于初學者能夠在脫離辦公環境的地方（如地鐵、公交等），也能輕松閱讀。
讀者對象
源碼閱讀是一項苦差事，人力和時間成本都很高，尤其對于剛剛接觸Spark的人來說更是如此。本書盡可能保留源碼，使得分析過程不至于產生跳躍感，目的是降低大多數人的學習門檻。如果你是從事IT工作1∼3年的新人或者希望開始學習Spark的核心知識，本書非常適合你。如果你已經對Spark有所了解或者已經使用它，還想進一步提高自己，那么本書更適合你。如果你是一個開發新手，對Java、Linux等基礎知識還不是很了解的話，本書可能不太適合你。如果你已經對Spark有深入的研究，本書也許可以作為你的參考資料。
總體來說，本書適合以下人群：
已經了解過Spark，但還想深入理解Spark實現原理的人；
大數據技術愛好者；
對性能優化和部署方案感興趣的運維工程師與架構師；
開源代碼愛好者，喜歡研究源碼的同學可以通過本書學到一些閱讀源碼的方式、方法。
本書不會教你如何開發Spark應用程序，而只拿word count的經典例子做演示。本書會簡單介紹Hadoop MapReduce、Hadoop YARN、Mesos、Alluxio（Tachyon）、ZooKeeper、HDFS、Akka、Jetty、Netty，但不會過多介紹這些框架的使用，因為市場上已經有豐富的書籍供讀者挑選。本書也不會過多介紹Scala、Java、Shell的語法，讀者可以在市場上選擇適合自己的書籍閱讀。本書將無比適合那些想要破解“潘多拉魔盒”的人！
如何閱讀本書
本書一共有10章內容，主要包括以下部分。
準備部分（第1∼2章）：簡單介紹了Spark的環境搭建和基本原理，幫助讀者了解一些背景知識。
基礎部分（第3∼5章）：介紹Spark的基礎設施、SparkContext的初始化、Spark執行環境等內容。
核心部分（第6∼9章）：這是Spark最為核心的部分，包括存儲體系、調度系統、計算引擎、部署模式等。
API部分（第10章）：這部分主要對Spark的新舊API進行對比，對新API進行介紹。
本書最后的附錄中還包括一些內容：附錄A介紹的是Spark中最常用的工具類Utils；附錄B是Akka的簡介；附錄C為Jetty的簡介和工具類JettyUtils的介紹；附錄D為Metrics庫的簡介和Metrics中部分API的介紹；附錄E演示了Hadoop 1.0版本中的word count例子；附錄F 介紹了工具類CommandUtils的常用方法；附錄G是關于Netty的簡介和工具類NettyUtils的介紹；附錄H是對Spark中的RPC工具類RpcUtils的介紹。
為了降低讀者閱讀理解Spark源碼的門檻，本書盡可能保留源碼實現。本書以Spark 2.1.0版本為主，有興趣的讀者也可按照本書的方式，閱讀Spark的最新源碼。
勘誤
本書內容很多，限于筆者水平有限，書中內容難免有錯誤之處。如果你對本書有任何問題或者意見，都可以通過郵箱beliefer@163.com或者博客http://blog.csdn.net/beliefer聯系我，給我提交你的建議或者想法，我將懷著一顆謙卑之心與大家共同進步。
致謝
感謝我們生活在信息時代，讓我們有機會接觸互聯網與大數據；感謝父母多年來在學習、工作及生活上的幫助與支持；感謝妻子在生活中的照顧和謙讓。
感謝高婧雅編輯給予本書出版的大力支持與幫助。
感謝我在大數據路上的領路人—和仲；感謝熱衷于技術的王歡對本書內容提出的寶貴建議；感謝對本書內容進行審閱的余堯堯和馬曉波；感謝對本書內容有過幫助的讀者朋友們。
耿嘉安 |
內容簡介：內容簡介
多位專家聯袂推薦，360大數據專家撰寫，基于Spark 2.1.0剖析架構與實現精髓。細化到方法級，提煉出多個流程圖，立體呈現架構、環境、調度、存儲、計算、部署、API七大核心設計。本書一共有10章內容，主要包括以下部分。
準備部分（第1∼2章）：簡單介紹了Spark的環境搭建和基本原理。本部分通過詳盡的描述，有效降低了讀者進入Spark世界的門檻，同時能對Spark背景知識及整體設計有宏觀的認識。
基礎部分（第3∼5章）：介紹Spark的基礎設施（包括配置、RPC、度量等）、SparkContext的初始化、Spark執行所需要的環境等內容。經過此部分的學習，將能夠對RPC框架的設計、執行環境的功能有深入的理解，這也是對核心內容了解的前提。
核心部分（第6∼9章）：為Spark最核心的部分，包括存儲體系、調度系統、計算引擎、部署模式等。通過本部分的學習，讀者將充分了解Spark的數據處理體系細節，能夠對Spark核心功能進行擴展、性能優化以及對線上問題進行精準排查。
API部分（第10章）：這部分主要對Spark的新老API進行對比，對新API進行簡單介紹。 |
目錄：本書贊譽
前言
第1章 環境準備 ········································1
1.1 運行環境準備 ···········································2
1.1.1 安裝JDK ·········································2
1.1.2 安裝Scala ········································2
1.1.3 安裝Spark ·······································3
1.2 Spark初體驗 ···································4
1.2.1 運行spark-shell ·······························4
1.2.2 執行word count ······························5
1.2.3 剖析spark-shell ·······························9
1.3 閱讀環境準備 ·········································14
1.3.1 安裝SBT ·······································15
1.3.2 安裝Git ·········································15
1.3.3 安裝Eclipse Scala IDE插件 ········15
1.4 Spark源碼編譯與調試 ·························17
1.5 小結 ···························23
第2章 設計理念與基本架構 ···············24
2.1 初識Spark ··································25
2.1.1 Hadoop MRv1的局限···················25
2.1.2 Spark的特點 ·································26
2.1.3 Spark使用場景 ·····························28
2.2 Spark基礎知識 ······································29
2.3 Spark基本設計思想 ·····························31
2.3.1 Spark模塊設計 ·····························32
2.3.2 Spark模型設計 ·····························34
2.4 Spark基本架構 ···································36
2.5 小結 ·································38
第3章 Spark基礎設施 ·························39
3.1 Spark配置 ········································40
3.1.1 系統屬性中的配置 ·······················40
3.1.2 使用SparkConf配置的API ·········41
3.1.3 克隆SparkConf配置 ····················42
3.2 Spark內置RPC框架 ····························42
3.2.1 RPC配置TransportConf ··············45
3.2.2 RPC客戶端工廠Transport- ClientFactory ·······················47
3.2.3 RPC服務端TransportServer ········53
3.2.4 管道初始化 ···································56
3.2.5 TransportChannelHandler詳解 ·····57
3.2.6 服務端RpcHandler詳解 ··············63
3.2.7 服務端引導程序Transport-ServerBootstrap ·····················68
3.2.8 客戶端TransportClient詳解 ········71
3.3 事件總線 ····································78
3.3.1 ListenerBus的繼承體系 ···············79
3.3.2 SparkListenerBus詳解 ··················80
3.3.3 LiveListenerBus詳解 ····················83
3.4 度量系統 ···········································87
3.4.1 Source繼承體系 ···························87
3.4.2 Sink繼承體系 ·······························89
3.5 小結 ·········································92
第4章 SparkContext的初始化 ·········93
4.1 SparkContext概述 ·································94
4.2 創建Spark環境 ·····································97
4.3 SparkUI的實現 ····································100
4.3.1 SparkUI概述 ·······························100
4.3.2 WebUI框架體系 ·························102
4.3.3 創建SparkUI ·······························107
4.4 創建心跳接收器 ··································111
4.5 創建和啟動調度系統··························112
4.6 初始化塊管理器BlockManager ·······114
4.7 啟動度量系統 ·······························114
4.8 創建事件日志監聽器··························115
4.9 創建和啟動ExecutorAllocation-Manager ··························116
4.10 ContextCleaner的創建與啟動 ········120
4.10.1 創建ContextCleaner ·················120
4.10.2 啟動ContextCleaner ·················120
4.11 額外的SparkListener與啟動事件總線 ··························122
4.12 Spark環境更新 ··································123
4.13 SparkContext初始化的收尾 ···········127
4.14 SparkContext提供的常用方法 ·······128
4.15 SparkContext的伴生對象················130
4.16 小結 ····································131
第5章 Spark執行環境 ························132
5.1 SparkEnv概述 ·································133
5.2 安全管理器SecurityManager ············133
5.3 RPC環境 ·········································135
5.3.1 RPC端點RpcEndpoint ···············136
5.3.2 RPC端點引用RpcEndpointRef ···139
5.3.3 創建傳輸上下文TransportConf ···142
5.3.4 消息調度器Dispatcher ···············142
5.3.5 創建傳輸上下文Transport-Context ·························154
5.3.6 創建傳輸客戶端工廠Transport-ClientFactory ····················159
5.3.7 創建TransportServer ···················160
5.3.8 客戶端請求發送 ·························162
5.3.9 NettyRpcEnv中的常用方法 ·······173
5.4 序列化管理器SerializerManager ·····175
5.5 廣播管理器BroadcastManager ·········178
5.6 map任務輸出跟蹤器 ··························185
5.6.1 MapOutputTracker的實現 ··········187
5.6.2 MapOutputTrackerMaster的實現原理 ·······················191
5.7 構建存儲體系 ·······································199
5.8 創建度量系統 ·······································201
5.8.1 MetricsCon?g詳解 ·····················203
5.8.2 MetricsSystem中的常用方法 ····207
5.8.3 啟動MetricsSystem ····················209
5.9 輸出提交協調器 ··································211
5.9.1 OutputCommitCoordinator-Endpoint的實現 ··················211
5.9.2 OutputCommitCoordinator的實現 ··························212
5.9.3 OutputCommitCoordinator的工作原理 ························216
5.10 創建SparkEnv ····································217
5.11 小結 ·····································217
第6章 存儲體系 ·····································219
6.1 存儲體系概述 ·······································220
6.1.1 存儲體系架構 ·····························220
6.1.2 基本概念 ·····································222
6.2 Block信息管理器 ································227
6.2.1 Block鎖的基本概念 ···················227
6.2.2 Block鎖的實現 ···························229
6.3 磁盤Block管理器 ······························234
6.3.1 本地目錄結構 ·····························234
6.3.2 DiskBlockManager提供的方法 ···························236
6.4 磁盤存儲DiskStore ·····························239
6.5 內存管理器 ·····································242
6.5.1 內存池模型 ·································243
6.5.2 StorageMemoryPool詳解 ···········244
6.5.3 MemoryManager模型 ················247
6.5.4 Uni?edMemoryManager詳解 ····250
6.6 內存存儲MemoryStore ······················252
6.6.1 MemoryStore的內存模型 ··········253
6.6.2 MemoryStore提供的方法 ··········255
6.7 塊管理器BlockManager ····················265
6.7.1 BlockManager的初始化 ·············265
6.7.2 BlockManager提供的方法 ·········266
6.8 BlockManagerMaster對Block-Manager的管理 ·················285
6.8.1 BlockManagerMaster的職責 ······285
6.8.2 BlockManagerMasterEndpoint詳解 ·································286
6.8.3 BlockManagerSlaveEndpoint詳解 ·····························289
6.9 Block傳輸服務 ····································290
6.9.1 初始化NettyBlockTransfer-Service ···························291
6.9.2 NettyBlockRpcServer詳解 ·········292
6.9.3 Shuf?e客戶端 ·····························296
6.10 DiskBlockObjectWriter詳解 ···········305
6.11 小結 ·······································308
第7章 調度系統 ·····································309
7.1 調度系統概述 ·······································310
7.2 RDD詳解 ·····································312
7.2.1 為什么需要RDD ························312
7.2.2 RDD實現的初次分析 ················313
7.2.3 RDD依賴 ····································316
7.2.4 分區計算器Partitioner················318
7.2.5 RDDInfo ······································320
7.3 Stage詳解 ········································321
7.3.1 ResultStage的實現 ·····················322
7.3.2 Shuf?eMapStage的實現 ·············323
7.3.3 StageInfo ······································324
7.4 面向DAG的調度器DAGScheduler ···326
7.4.1 JobListener與JobWaiter ·············326
7.4.2 ActiveJob詳解 ····························328
7.4.3 DAGSchedulerEventProcessLoop的簡要介紹 ·······················328
7.4.4 DAGScheduler的組成 ················329
7.4.5 DAGScheduler提供的常用方法 ···330
7.4.6 DAGScheduler與Job的提交 ····334
7.4.7 構建Stage····································337
7.4.8 提交ResultStage ························341
7.4.9 提交還未計算的Task ·················343
7.4.10 DAGScheduler的調度流程 ······347
7.4.11 Task執行結果的處理 ··············348
7.5 調度池Pool ······································351
7.5.1 調度算法 ·······························352
7.5.2 Pool的實現 ·································354
7.5.3 調度池構建器 ·····························357
7.6 任務集合管理器TaskSetManager ···363
7.6.1 Task集合 ·····································363
7.6.2 TaskSetManager的成員屬性 ······364
7.6.3 調度池與推斷執行 ·····················366
7.6.4 Task本地性 ·································370
7.6.5 TaskSetManager的常用方法 ······373
7.7 運行器后端接口LauncherBackend ···383
7.7.1 BackendConnection的實現 ········384
7.7.2 LauncherBackend的實現 ···········386
7.8 調度后端接口SchedulerBackend ····389
7.8.1 SchedulerBackend的定義 ··········389
7.8.2 LocalSchedulerBackend的實現分析 ································390
7.9 任務結果獲取器TaskResultGetter ···394
7.9.1 處理成功的Task ·························394
7.9.2 處理失敗的Task ·························396
7.10 任務調度器TaskScheduler ··············397
7.10.1 TaskSchedulerImpl的屬性 ·····397
7.10.2 TaskSchedulerImpl的初始化 ···399
7.10.3 TaskSchedulerImpl的啟動 ·····399
7.10.4 TaskSchedulerImpl與Task的提交 ·······················400
7.10.5 TaskSchedulerImpl與資源分配 ···························402
7.10.6 TaskSchedulerImpl的調度流程 ······························405
7.10.7 TaskSchedulerImpl對執行結果的處理 ·····························406
7.10.8 TaskSchedulerImpl的常用方法 ···409
7.11 小結 ·······································412
第8章 計算引擎 ·····································413
8.1 計算引擎概述 ·······································414
8.2 內存管理器與執行內存 ·····················417
8.2.1 ExecutionMemoryPool詳解 ·······417
8.2.2 MemoryManager模型與執行內存 ··························420
8.2.3 Uni?edMemoryManager與執行內存 ·······················421
8.3 內存管理器與Tungsten ·····················423
8.3.1 MemoryBlock詳解 ·····················423
8.3.2 MemoryManager模型與Tungsten ···························425
8.3.3 Tungsten的內存分配器 ··············425
8.4 任務內存管理器 ··································431
8.4.1 TaskMemoryManager詳解 ·········431
8.4.2 內存消費者 ·······················439
8.4.3 執行內存整體架構 ·····················441
8.5 Task詳解 ······································443
8.5.1 任務上下文TaskContext ············443
8.5.2 Task的定義 ·································446
8.5.3 Shuf?eMapTask的實現 ··············449
8.5.4 ResultTask的實現 ·······················450
8.6 IndexShuf?eBlockResolver詳解 ······451
8.7 采樣與估算 ···········································455
8.7.1 SizeTracker的實現分析 ·············455
8.7.2 SizeTracker的工作原理 ·············457
8.8 特質WritablePartitionedPair- Collection ······················458
8.9 AppendOnlyMap的實現分析 ···········460
8.9.1 AppendOnlyMap的容量增長 ····461
8.9.2 AppendOnlyMap的數據更新 ····462
8.9.3 AppendOnlyMap的緩存聚合算法 ·····························464
8.9.4 AppendOnlyMap的內置排序 ····466
8.9.5 AppendOnlyMap的擴展 ············467
8.10 PartitionedPairBuffer的實現分析 ···469
8.10.1 PartitionedPairBuffer的容量增長 ······················469
8.10.2 PartitionedPairBuffer的插入 ···470
8.10.3 PartitionedPairBuffer的迭代器 ···471
8.11 外部排序器 ·········································472
8.11.1 ExternalSorter詳解 ·················473
8.11.2 Shuf?eExternalSorter詳解 ······487
8.12 Shuf?e管理器 ····································490
8.12.1 Shuf?eWriter詳解 ··················491
8.12.2 Shuf?eBlockFetcherIterator詳解 ······························502
8.12.3 BlockStoreShuf?eReader詳解 ···510
8.12.4 SortShuf?eManager詳解 ········513
8.13 map端與reduce端的Shuf?e組合 ······························516
8.14 小結 ·········································519
第9章 部署模式 ········································520
9.1 心跳接收器HeartbeatReceiver ·········521
9.2 Executor的實現分析 ··························527
9.2.1 Executor的心跳報告 ··················528
9.2.2 運行Task ·····································530
9.3 local部署模式 ······································535
9.4 持久化引擎PersistenceEngine ··········537
9.4.1 基于文件系統的持久化引擎 ·····539
9.4.2 基于ZooKeeper的持久化引擎 ···541
9.5 領導選舉代理 ·······································542
9.6 Master詳解 ···········································546
9.6.1 啟動Master ·································549
9.6.2 檢查Worker超時························553
9.6.3 被選舉為領導時的處理 ·············554
9.6.4 一級資源調度 ·····························558
9.6.5 注冊Worker·································568
9.6.6 更新Worker的最新狀態············570
9.6.7 處理Worker的心跳····················570
9.6.8 注冊Application··························571
9.6.9 處理Executor的申請 ·················573
9.6.10 處理Executor的狀態變化 ·······573
9.6.11 Master的常用方法 ···················574
9.7 Worker詳解 ································578
9.7.1 啟動Worker·································581
9.7.2 向Master注冊Worker ···············584
9.7.3 向Master發送心跳 ····················589
9.7.4 Worker與領導選舉·····················591
9.7.5 運行Driver ··································593
9.7.6 運行Executor ······························594
9.7.7 處理Executor的狀態變化 ·········599
9.8 StandaloneAppClient實現 ·················600
9.8.1 ClientEndpoint的實現分析 ········601
9.8.2 StandaloneAppClient的實現分析 ······························606
9.9 StandaloneSchedulerBackend的實現分析 ························607
9.9.1 StandaloneSchedulerBackend的屬性 ····························607
9.9.2 DriverEndpoint的實現分析 ·······609
9.9.3 StandaloneSchedulerBackend的啟動 ··························614
9.9.4 StandaloneSchedulerBackend的停止 ·························617
9.9.5 StandaloneSchedulerBackend與資源分配 ················618
9.10 CoarseGrainedExecutorBackend詳解 ····························619
9.10.1 CoarseGrainedExecutorBackend進程 ··························620
9.10.2 CoarseGrainedExecutorBackend的功能分析 ·························622
9.11 local-cluster部署模式 ·······················625
9.11.1 啟動本地集群 ····························625
9.11.2 local-cluster部署模式的啟動過程 ·································627
9.11.3 local-cluster部署模式下Executor的分配過程 ·················628
9.11.4 local-cluster部署模式下的任務提交執行過程 ····························629
9.12 Standalone部署模式 ·························631
9.12.1 Standalone部署模式的啟動過程 ························632
9.12.2 Standalone部署模式下Executor的分配過程 ················634
9.12.3 Standalone部署模式的資源回收 ·····························635
9.12.4 Standalone部署模式的容錯機制 ······························636
9.13 其他部署方案 ·····································639
9.13.1 YARN·········································639
9.13.2 Mesos ·········································644
9.14 小結 ·······································646
第10章 Spark API ································647
10.1 基本概念·····································648
10.2 數據源DataSource ····························650
10.2.1 DataSourceRegister詳解 ··········650
10.2.2 DataSource詳解 ························651
10.3 檢查點的實現 ···································655
10.3.1 CheckpointRDD的實現············655
10.3.2 RDDCheckpointData的實現 ····660
10.3.3 ReliableRDDCheckpointData的實現 ························662
10.4 RDD的再次分析 ·······························663
10.4.1 轉換API ····································663
10.4.2 動作API ····································665
10.4.3 檢查點API的實現分析 ···········667
10.4.4 迭代計算 ···································669
10.5 數據集合Dataset ·······························671
10.6 DataFrameReader詳解 ·····················673
10.7 SparkSession詳解 ·····························676
10.7.1 SparkSession的構建器Builder ···676
10.7.2 SparkSession的API ·················679
10.8 word count例子 ·································679
10.8.1 Job準備階段 ·····························680
10.8.2 Job的提交與調度 ·····················685
10.9 小結 ········································689
附錄 ···········································690 |
| 序： |
