spark BlockDataManager 源码

2022-10-20
浏览 (378)

spark BlockDataManager 代码

文件路径：/core/src/main/scala/org/apache/spark/network/BlockDataManager.scala

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.spark.network

import scala.reflect.ClassTag

import org.apache.spark.TaskContext
import org.apache.spark.network.buffer.ManagedBuffer
import org.apache.spark.network.client.StreamCallbackWithID
import org.apache.spark.network.shuffle.checksum.Cause
import org.apache.spark.storage.{BlockId, StorageLevel}

private[spark]
trait BlockDataManager {

  /**
   * Diagnose the possible cause of the shuffle data corruption by verifying the shuffle checksums
   */
  def diagnoseShuffleBlockCorruption(
      blockId: BlockId,
      checksumByReader: Long,
      algorithm: String): Cause

  /**
   * Get the local directories that used by BlockManager to save the blocks to disk
   */
  def getLocalDiskDirs: Array[String]

  /**
   * Interface to get host-local shuffle block data. Throws an exception if the block cannot be
   * found or cannot be read successfully.
   */
  def getHostLocalShuffleData(blockId: BlockId, dirs: Array[String]): ManagedBuffer

  /**
   * Interface to get local block data. Throws an exception if the block cannot be found or
   * cannot be read successfully.
   */
  def getLocalBlockData(blockId: BlockId): ManagedBuffer

  /**
   * Put the block locally, using the given storage level.
   *
   * Returns true if the block was stored and false if the put operation failed or the block
   * already existed.
   */
  def putBlockData(
      blockId: BlockId,
      data: ManagedBuffer,
      level: StorageLevel,
      classTag: ClassTag[_]): Boolean

  /**
   * Put the given block that will be received as a stream.
   *
   * When this method is called, the block data itself is not available -- it will be passed to the
   * returned StreamCallbackWithID.
   */
  def putBlockDataAsStream(
      blockId: BlockId,
      level: StorageLevel,
      classTag: ClassTag[_]): StreamCallbackWithID

  /**
   * Release locks acquired by [[putBlockData()]] and [[getLocalBlockData()]].
   */
  def releaseLock(blockId: BlockId, taskContext: Option[TaskContext]): Unit
}

相关信息

spark 源码目录

相关文章

spark BlockTransferService 源码

0 赞

所属分类： 前端技术
本文标签：
版权声明： 原创文章如转载，请注明本文链接: https://www.seaxiang.com/blog/ff6402778e854ecaa5fdc8cb6b2f5e15

热门推荐

1、直接访问google.com
2、 - 优质文章
3、 gate.io
4、 harmony 鸿蒙hdc使用指导
5、 harmony 鸿蒙初识ArkTS语言
6、 harmony 鸿蒙ArkUI组件（ArkTS）开发常见问题
7、 flink kafka connector scan.startup.mode 的几个选项
8、 golang
9、 openharmony
10、 Vue中input框自动聚焦

Loading...